==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-8dc49bd56-qr24k Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Fri, 12 Apr 2024 21:01:24 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=8dc49bd56 skaffold.dev/run-id=378fbf6e-cf99-4f5f-928d-45050baebd01 Annotations: Status: Running IP: 10.106.42.165 IPs: IP: 10.106.42.165 Controlled By: ReplicaSet/lodemon-8dc49bd56 Containers: lodemon: Container ID: containerd://2611adbf40d7d7fc8aa95b19414dbcb2f4ac6640d117d77dba7d226f69a0d03f Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 12 Apr 2024 21:01:42 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pcb5r (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-pcb5r: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 02:05:10 INFO 02:05:10 INFO [loop_until]: kubectl --namespace=xlou top pods 02:05:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:05:10 INFO [loop_until]: OK (rc = 0) 02:05:10 DEBUG --- stdout --- 02:05:10 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10532m 8528Mi am-6d96779b5d-qbg7l 8482m 8460Mi am-6d96779b5d-zn69c 10235m 8405Mi ds-cts-0 9883m 13451Mi ds-cts-1 9929m 13445Mi ds-cts-2 9745m 13451Mi ds-idrepo-0 7536m 5593Mi ds-idrepo-1 3721m 5521Mi ds-idrepo-2 3617m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4753m 4064Mi 02:05:10 DEBUG --- stderr --- 02:05:10 DEBUG 02:05:16 INFO 02:05:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:05:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:05:16 INFO [loop_until]: OK (rc = 0) 02:05:16 DEBUG --- stdout --- 02:05:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4731Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1850Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8966m 30% 10337Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11900m 39% 3092Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10447m 34% 10208Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 4314Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11079m 37% 10375Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3880m 24% 6286Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9747m 61% 14290Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10013m 63% 14251Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3716m 23% 6680Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9881m 62% 14286Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7664m 48% 6366Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4844m 30% 7815Mi 13% 02:05:16 DEBUG --- stderr --- 02:05:16 DEBUG 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "0.021" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "1580.4666666666667" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "1573.9" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "ds-idrepo-1" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "ds-idrepo-0" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "0" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "ds-idrepo-1" 02:05:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:05:59 DEBUG }, 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "ds-cts-1" 02:05:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:05:59 DEBUG }, 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "0" 02:05:59 DEBUG "1580.5333333333333" 02:05:59 DEBUG ] 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "281787.5" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970359" 02:05:59 ERROR [http_cmd]: ERROR 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 ERROR --- status code --- 02:05:59 DEBUG --- status code --- 02:05:59 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 ERROR --- http response --- 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:05:59 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "pod": "ds-idrepo-1" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "pod": "ds-cts-1" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "29.8" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "instance": "10.142.0.107:9100" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "instance": "10.142.0.77:9100" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "46565694.63333333" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970359" 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970359" 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "instance": "10.142.0.107:9100" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "instance": "10.142.0.77:9100" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "536647650304" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: http status code OK 02:05:59 DEBUG --- status code --- 02:05:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:05:59 DEBUG --- http response --- 02:05:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:05:59 DEBUG ----- output ----- 02:05:59 DEBUG { 02:05:59 DEBUG "status": "success", 02:05:59 DEBUG "data": { 02:05:59 DEBUG "resultType": "vector", 02:05:59 DEBUG "result": [ 02:05:59 DEBUG { 02:05:59 DEBUG "metric": { 02:05:59 DEBUG "instance": "10.142.0.107:9100" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG [...] 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG 02:05:59 DEBUG "instance": "10.142.0.77:9100" 02:05:59 DEBUG }, 02:05:59 DEBUG "value": [ 02:05:59 DEBUG 1712970359, 02:05:59 DEBUG "46569433.46666667" 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG ] 02:05:59 DEBUG } 02:05:59 DEBUG } 02:05:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970359" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-idrepo-0" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970359, 02:06:00 DEBUG "0" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "1579.7" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "2925.494323735057" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "10624.16359672205" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-cts-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0.00046289166666666664" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-idrepo-0" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-cts-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "4.770294537689887" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970360" 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "6237839360" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "instance": "10.142.0.107:9100" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "instance": "10.142.0.77:9100" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0.17826666666666333" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "instance": "10.142.0.107:9100" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "instance": "10.142.0.77:9100" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0.06000000000000014" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "ds-idrepo-1" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "3.817047186865503" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970360" 02:06:00 INFO [http_cmd]: http status code OK 02:06:00 DEBUG --- status code --- 02:06:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:00 DEBUG --- http response --- 02:06:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:00 DEBUG ----- output ----- 02:06:00 DEBUG { 02:06:00 DEBUG "status": "success", 02:06:00 DEBUG "data": { 02:06:00 DEBUG "resultType": "vector", 02:06:00 DEBUG "result": [ 02:06:00 DEBUG { 02:06:00 DEBUG "metric": { 02:06:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG [...] 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG 02:06:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:00 DEBUG }, 02:06:00 DEBUG "value": [ 02:06:00 DEBUG 1712970360, 02:06:00 DEBUG "0" 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG ] 02:06:00 DEBUG } 02:06:00 DEBUG } 02:06:10 INFO 02:06:10 INFO [loop_until]: kubectl --namespace=xlou top pods 02:06:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:06:10 INFO [loop_until]: OK (rc = 0) 02:06:10 DEBUG --- stdout --- 02:06:10 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10861m 8529Mi am-6d96779b5d-qbg7l 8867m 8463Mi am-6d96779b5d-zn69c 10277m 8404Mi ds-cts-0 9722m 13449Mi ds-cts-1 9873m 13446Mi ds-cts-2 9612m 13450Mi ds-idrepo-0 7467m 5593Mi ds-idrepo-1 3751m 5521Mi ds-idrepo-2 3765m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4661m 4063Mi 02:06:10 DEBUG --- stderr --- 02:06:10 DEBUG 02:06:16 INFO 02:06:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:06:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:06:16 INFO [loop_until]: OK (rc = 0) 02:06:16 DEBUG --- stdout --- 02:06:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 4739Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1857Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9086m 30% 10345Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12367m 41% 3079Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10556m 35% 10199Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 4309Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11128m 37% 10368Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3906m 24% 6284Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9776m 61% 14296Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 9978m 62% 14256Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3792m 23% 6677Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9536m 60% 14294Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7636m 48% 6365Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4708m 29% 7797Mi 13% 02:06:16 DEBUG --- stderr --- 02:06:16 DEBUG 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "1569.6333333333332" 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "0.021" 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "1558.3" 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "ds-idrepo-1" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "ds-idrepo-0" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "0" 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "ds-idrepo-1" 02:06:59 DEBUG }, 02:06:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:59 DEBUG "value": [ 02:06:59 DEBUG }, 02:06:59 DEBUG 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "ds-cts-1" 02:06:59 DEBUG 02:06:59 DEBUG }, 02:06:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:59 DEBUG "value": [ 02:06:59 DEBUG }, 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG "0" 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG ] 02:06:59 DEBUG "1569.5333333333333" 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970419" 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "282498.5" 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: http status code OK 02:06:59 DEBUG --- status code --- 02:06:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:06:59 DEBUG --- http response --- 02:06:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:06:59 DEBUG ----- output ----- 02:06:59 DEBUG { 02:06:59 DEBUG "status": "success", 02:06:59 DEBUG "data": { 02:06:59 DEBUG "resultType": "vector", 02:06:59 DEBUG "result": [ 02:06:59 DEBUG { 02:06:59 DEBUG "metric": { 02:06:59 DEBUG "pod": "ds-idrepo-1" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG [...] 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG 02:06:59 DEBUG "pod": "ds-cts-1" 02:06:59 DEBUG }, 02:06:59 DEBUG "value": [ 02:06:59 DEBUG 1712970419, 02:06:59 DEBUG "29.633333333333333" 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG ] 02:06:59 DEBUG } 02:06:59 DEBUG } 02:06:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970419" 02:06:59 ERROR [http_cmd]: ERROR 02:06:59 ERROR --- status code --- 02:06:59 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:06:59 ERROR --- http response --- 02:06:59 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:06:59 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970420" 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "instance": "10.142.0.107:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "instance": "10.142.0.77:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "46454824" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "ds-idrepo-1" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "ds-idrepo-0" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "instance": "10.142.0.107:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "instance": "10.142.0.77:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "538894205952" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "instance": "10.142.0.107:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "instance": "10.142.0.77:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "46457949.56666666" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "1569.4666666666667" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "ds-idrepo-1" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "ds-cts-1" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0.00036173333333333336" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "977.2277647569279" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "34831205.88044443" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "ds-idrepo-1" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "ds-idrepo-0" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "ds-idrepo-1" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "ds-cts-1" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "4.716552654279094" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970420" 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "6237151232" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "instance": "10.142.0.107:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "instance": "10.142.0.77:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0.09166666666666382" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "instance": "10.142.0.107:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "instance": "10.142.0.77:9100" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0.03433333333333148" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0.0046531829101174385" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970420" 02:07:00 INFO [http_cmd]: http status code OK 02:07:00 DEBUG --- status code --- 02:07:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:00 DEBUG --- http response --- 02:07:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:00 DEBUG ----- output ----- 02:07:00 DEBUG { 02:07:00 DEBUG "status": "success", 02:07:00 DEBUG "data": { 02:07:00 DEBUG "resultType": "vector", 02:07:00 DEBUG "result": [ 02:07:00 DEBUG { 02:07:00 DEBUG "metric": { 02:07:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG [...] 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG 02:07:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:07:00 DEBUG }, 02:07:00 DEBUG "value": [ 02:07:00 DEBUG 1712970420, 02:07:00 DEBUG "0" 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG ] 02:07:00 DEBUG } 02:07:00 DEBUG } 02:07:11 INFO 02:07:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:07:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:07:11 INFO [loop_until]: OK (rc = 0) 02:07:11 DEBUG --- stdout --- 02:07:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10199m 8529Mi am-6d96779b5d-qbg7l 8602m 8463Mi am-6d96779b5d-zn69c 10332m 8405Mi ds-cts-0 9713m 13451Mi ds-cts-1 9867m 13447Mi ds-cts-2 9446m 13450Mi ds-idrepo-0 7387m 5593Mi ds-idrepo-1 3776m 5522Mi ds-idrepo-2 3764m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4698m 4064Mi 02:07:11 DEBUG --- stderr --- 02:07:11 DEBUG 02:07:16 INFO 02:07:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:07:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:07:16 INFO [loop_until]: OK (rc = 0) 02:07:16 DEBUG --- stdout --- 02:07:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4726Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1841Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8914m 29% 10331Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12289m 41% 3073Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10515m 35% 10197Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 4330Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10983m 36% 10370Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3916m 24% 6283Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9633m 60% 14281Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10032m 63% 14262Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3644m 22% 6678Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9735m 61% 14286Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7645m 48% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4837m 30% 7794Mi 13% 02:07:16 DEBUG --- stderr --- 02:07:16 DEBUG 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 INFO [http_cmd]: http status code OK 02:07:59 INFO [http_cmd]: http status code OK 02:07:59 DEBUG --- status code --- 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 DEBUG --- status code --- 02:07:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:59 DEBUG --- http response --- 02:07:59 DEBUG --- http response --- 02:07:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:59 DEBUG ----- output ----- 02:07:59 DEBUG ----- output ----- 02:07:59 DEBUG { 02:07:59 DEBUG { 02:07:59 DEBUG "status": "success", 02:07:59 DEBUG "status": "success", 02:07:59 DEBUG "data": { 02:07:59 DEBUG "data": { 02:07:59 DEBUG "resultType": "vector", 02:07:59 DEBUG "resultType": "vector", 02:07:59 DEBUG "result": [ 02:07:59 DEBUG "result": [ 02:07:59 DEBUG { 02:07:59 DEBUG { 02:07:59 DEBUG "metric": { 02:07:59 DEBUG "metric": { 02:07:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:07:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:07:59 DEBUG }, 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG [...] 02:07:59 DEBUG [...] 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:07:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:07:59 DEBUG }, 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 1712970479, 02:07:59 DEBUG 1712970479, 02:07:59 DEBUG "1575.1666666666667" 02:07:59 DEBUG "0.021" 02:07:59 DEBUG ] 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 DEBUG ] 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 INFO [http_cmd]: http status code OK 02:07:59 DEBUG --- status code --- 02:07:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:59 DEBUG --- http response --- 02:07:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:59 DEBUG ----- output ----- 02:07:59 DEBUG { 02:07:59 DEBUG "status": "success", 02:07:59 DEBUG "data": { 02:07:59 DEBUG "resultType": "vector", 02:07:59 DEBUG "result": [ 02:07:59 DEBUG { 02:07:59 DEBUG "metric": { 02:07:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG [...] 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 1712970479, 02:07:59 DEBUG "1570.0333333333333" 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 INFO [http_cmd]: http status code OK 02:07:59 DEBUG --- status code --- 02:07:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:59 DEBUG --- http response --- 02:07:59 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 INFO [http_cmd]: http status code OK 02:07:59 DEBUG --- status code --- 02:07:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:59 DEBUG --- http response --- 02:07:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:59 DEBUG ----- output ----- 02:07:59 DEBUG { 02:07:59 DEBUG "status": "success", 02:07:59 DEBUG "data": { 02:07:59 DEBUG "resultType": "vector", 02:07:59 DEBUG "result": [ 02:07:59 DEBUG { 02:07:59 DEBUG "metric": { 02:07:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG [...] 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 1712970479, 02:07:59 DEBUG "283139.5" 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 INFO [http_cmd]: http status code OK 02:07:59 DEBUG --- status code --- 02:07:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:07:59 DEBUG --- http response --- 02:07:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:07:59 DEBUG ----- output ----- 02:07:59 DEBUG { 02:07:59 DEBUG "status": "success", 02:07:59 DEBUG "data": { 02:07:59 DEBUG "resultType": "vector", 02:07:59 DEBUG "result": [ 02:07:59 DEBUG { 02:07:59 DEBUG "metric": { 02:07:59 DEBUG "pod": "ds-idrepo-1" 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG [...] 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG 02:07:59 DEBUG "pod": "ds-idrepo-0" 02:07:59 DEBUG }, 02:07:59 DEBUG "value": [ 02:07:59 DEBUG 1712970479, 02:07:59 DEBUG "0" 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG ] 02:07:59 DEBUG } 02:07:59 DEBUG } 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:07:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970479" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "ds-idrepo-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "ds-cts-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970479, 02:08:00 DEBUG "0" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970479, 02:08:00 DEBUG "1575.2333333333333" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970480" 02:08:00 ERROR [http_cmd]: ERROR 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 ERROR --- status code --- 02:08:00 DEBUG --- status code --- 02:08:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 ERROR --- http response --- 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:08:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "ds-idrepo-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "ds-cts-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "29.599999999999998" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "instance": "10.142.0.107:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "instance": "10.142.0.77:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "541175974912" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970480" 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "instance": "10.142.0.107:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "instance": "10.142.0.77:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "47097521.13333333" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "ds-idrepo-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "ds-idrepo-0" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "instance": "10.142.0.107:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "instance": "10.142.0.77:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "47100518.666666664" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "1575.3" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "ds-idrepo-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "ds-cts-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0.00006666666666666662" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "9973.073617874534" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "586.6513892867373" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "ds-idrepo-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "ds-idrepo-0" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "ds-idrepo-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "ds-cts-1" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "4.741987211435113" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970480" 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "6237282304" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "instance": "10.142.0.107:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "instance": "10.142.0.77:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0.18376666666666117" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970480" 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "instance": "10.142.0.107:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "instance": "10.142.0.77:9100" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0.06966666666666607" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0.015171333428817402" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970480" 02:08:00 INFO [http_cmd]: http status code OK 02:08:00 DEBUG --- status code --- 02:08:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:00 DEBUG --- http response --- 02:08:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:00 DEBUG ----- output ----- 02:08:00 DEBUG { 02:08:00 DEBUG "status": "success", 02:08:00 DEBUG "data": { 02:08:00 DEBUG "resultType": "vector", 02:08:00 DEBUG "result": [ 02:08:00 DEBUG { 02:08:00 DEBUG "metric": { 02:08:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG [...] 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG 02:08:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:08:00 DEBUG }, 02:08:00 DEBUG "value": [ 02:08:00 DEBUG 1712970480, 02:08:00 DEBUG "0" 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG ] 02:08:00 DEBUG } 02:08:00 DEBUG } 02:08:11 INFO 02:08:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:08:11 INFO [loop_until]: OK (rc = 0) 02:08:11 DEBUG --- stdout --- 02:08:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10599m 8529Mi am-6d96779b5d-qbg7l 8614m 8462Mi am-6d96779b5d-zn69c 10371m 8417Mi ds-cts-0 9567m 13450Mi ds-cts-1 9566m 13451Mi ds-cts-2 9791m 13450Mi ds-idrepo-0 7538m 5593Mi ds-idrepo-1 3847m 5522Mi ds-idrepo-2 3686m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4553m 4063Mi 02:08:11 DEBUG --- stderr --- 02:08:11 DEBUG 02:08:16 INFO 02:08:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:08:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:08:16 INFO [loop_until]: OK (rc = 0) 02:08:16 DEBUG --- stdout --- 02:08:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 4740Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 1860Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9005m 30% 10326Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12325m 41% 3069Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10631m 35% 10218Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 4305Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11032m 36% 10372Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3761m 23% 6284Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9814m 61% 14293Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 9848m 61% 14279Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3787m 23% 6680Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10011m 63% 14296Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7642m 48% 6367Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4864m 30% 7799Mi 13% 02:08:16 DEBUG --- stderr --- 02:08:16 DEBUG 02:08:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970539" 02:08:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970539" 02:08:59 INFO [http_cmd]: http status code OK 02:08:59 DEBUG --- status code --- 02:08:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:59 DEBUG --- http response --- 02:08:59 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:08:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970539" 02:08:59 INFO [http_cmd]: http status code OK 02:08:59 DEBUG --- status code --- 02:08:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:59 DEBUG --- http response --- 02:08:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:59 DEBUG ----- output ----- 02:08:59 DEBUG { 02:08:59 DEBUG "status": "success", 02:08:59 DEBUG "data": { 02:08:59 DEBUG "resultType": "vector", 02:08:59 DEBUG "result": [ 02:08:59 DEBUG { 02:08:59 DEBUG "metric": { 02:08:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:08:59 DEBUG }, 02:08:59 DEBUG "value": [ 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG [...] 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:08:59 DEBUG }, 02:08:59 DEBUG "value": [ 02:08:59 DEBUG 1712970539, 02:08:59 DEBUG "1579.2333333333333" 02:08:59 DEBUG ] 02:08:59 DEBUG } 02:08:59 DEBUG ] 02:08:59 DEBUG } 02:08:59 DEBUG } 02:08:59 INFO [http_cmd]: http status code OK 02:08:59 DEBUG --- status code --- 02:08:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:59 DEBUG --- http response --- 02:08:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:59 DEBUG ----- output ----- 02:08:59 DEBUG { 02:08:59 DEBUG "status": "success", 02:08:59 DEBUG "data": { 02:08:59 DEBUG "resultType": "vector", 02:08:59 DEBUG "result": [ 02:08:59 DEBUG { 02:08:59 DEBUG "metric": { 02:08:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:08:59 DEBUG }, 02:08:59 DEBUG "value": [ 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG [...] 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:08:59 DEBUG }, 02:08:59 DEBUG "value": [ 02:08:59 DEBUG 1712970539, 02:08:59 DEBUG "1566.6333333333332" 02:08:59 DEBUG ] 02:08:59 DEBUG } 02:08:59 DEBUG ] 02:08:59 DEBUG } 02:08:59 DEBUG } 02:08:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970539" 02:08:59 INFO [http_cmd]: http status code OK 02:08:59 DEBUG --- status code --- 02:08:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:08:59 DEBUG --- http response --- 02:08:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:08:59 DEBUG ----- output ----- 02:08:59 DEBUG { 02:08:59 DEBUG "status": "success", 02:08:59 DEBUG "data": { 02:08:59 DEBUG "resultType": "vector", 02:08:59 DEBUG "result": [ 02:08:59 DEBUG { 02:08:59 DEBUG "metric": { 02:08:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:08:59 DEBUG }, 02:08:59 DEBUG "value": [ 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG [...] 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG 02:08:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:08:59 DEBUG }, 02:08:59 DEBUG "value": [ 02:08:59 DEBUG 1712970539, 02:08:59 DEBUG "0.021" 02:08:59 DEBUG ] 02:08:59 DEBUG } 02:08:59 DEBUG ] 02:08:59 DEBUG } 02:08:59 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "1566.4333333333334" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "283192" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-idrepo-0" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "instance": "10.142.0.107:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "instance": "10.142.0.77:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "543245184000" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-cts-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "29.599999999999998" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-cts-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970540" 02:09:00 ERROR [http_cmd]: ERROR 02:09:00 ERROR --- status code --- 02:09:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:09:00 ERROR --- http response --- 02:09:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:09:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "instance": "10.142.0.107:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "instance": "10.142.0.77:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "46390488.3" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-idrepo-0" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "instance": "10.142.0.107:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "instance": "10.142.0.77:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "46394052.7" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "1566.8333333333333" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-cts-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0.0011637499999999999" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-cts-2" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "27210010.134643734" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-idrepo-0" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "ds-idrepo-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-cts-1" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "4.726398928792937" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "6238343168" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "instance": "10.142.0.107:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "instance": "10.142.0.77:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0.177833333333335" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "ds-cts-2" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "14747833.968659567" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970540" 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "instance": "10.142.0.107:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "instance": "10.142.0.77:9100" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0.07033333333333426" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970540" 02:09:00 INFO [http_cmd]: http status code OK 02:09:00 DEBUG --- status code --- 02:09:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:00 DEBUG --- http response --- 02:09:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:00 DEBUG ----- output ----- 02:09:00 DEBUG { 02:09:00 DEBUG "status": "success", 02:09:00 DEBUG "data": { 02:09:00 DEBUG "resultType": "vector", 02:09:00 DEBUG "result": [ 02:09:00 DEBUG { 02:09:00 DEBUG "metric": { 02:09:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG [...] 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG 02:09:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:00 DEBUG }, 02:09:00 DEBUG "value": [ 02:09:00 DEBUG 1712970540, 02:09:00 DEBUG "0" 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG ] 02:09:00 DEBUG } 02:09:00 DEBUG } 02:09:11 INFO 02:09:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:09:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:09:11 INFO [loop_until]: OK (rc = 0) 02:09:11 DEBUG --- stdout --- 02:09:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10316m 8530Mi am-6d96779b5d-qbg7l 8827m 8460Mi am-6d96779b5d-zn69c 10130m 8405Mi ds-cts-0 9738m 13449Mi ds-cts-1 10015m 13454Mi ds-cts-2 9356m 13451Mi ds-idrepo-0 7473m 5593Mi ds-idrepo-1 3794m 5522Mi ds-idrepo-2 3703m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1094Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4768m 4063Mi 02:09:11 DEBUG --- stderr --- 02:09:11 DEBUG 02:09:16 INFO 02:09:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:09:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:09:16 INFO [loop_until]: OK (rc = 0) 02:09:16 DEBUG --- stdout --- 02:09:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4747Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1851Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9083m 30% 10331Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12321m 41% 3065Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10490m 35% 10203Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 4312Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11099m 37% 10375Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3928m 24% 6285Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9689m 60% 14291Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10152m 63% 14265Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3755m 23% 6679Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9821m 61% 14308Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7658m 48% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4824m 30% 7798Mi 13% 02:09:16 DEBUG --- stderr --- 02:09:16 DEBUG 02:09:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970599" 02:09:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970599" 02:09:59 INFO [http_cmd]: http status code OK 02:09:59 DEBUG --- status code --- 02:09:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:59 DEBUG --- http response --- 02:09:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:59 DEBUG ----- output ----- 02:09:59 DEBUG { 02:09:59 DEBUG "status": "success", 02:09:59 DEBUG "data": { 02:09:59 DEBUG "resultType": "vector", 02:09:59 DEBUG "result": [ 02:09:59 DEBUG { 02:09:59 DEBUG "metric": { 02:09:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:59 DEBUG }, 02:09:59 DEBUG "value": [ 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG [...] 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:59 DEBUG }, 02:09:59 DEBUG "value": [ 02:09:59 DEBUG 1712970599, 02:09:59 DEBUG "0.021" 02:09:59 DEBUG ] 02:09:59 DEBUG } 02:09:59 DEBUG ] 02:09:59 DEBUG } 02:09:59 DEBUG } 02:09:59 INFO [http_cmd]: http status code OK 02:09:59 DEBUG --- status code --- 02:09:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:59 DEBUG --- http response --- 02:09:59 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:09:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970599" 02:09:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970599" 02:09:59 INFO [http_cmd]: http status code OK 02:09:59 INFO [http_cmd]: http status code OK 02:09:59 DEBUG --- status code --- 02:09:59 DEBUG --- status code --- 02:09:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:09:59 DEBUG --- http response --- 02:09:59 DEBUG --- http response --- 02:09:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:59 DEBUG ----- output ----- 02:09:59 DEBUG { 02:09:59 DEBUG "status": "success", 02:09:59 DEBUG "data": { 02:09:59 DEBUG "resultType": "vector", 02:09:59 DEBUG "result": [ 02:09:59 DEBUG { 02:09:59 DEBUG "metric": { 02:09:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:59 DEBUG }, 02:09:59 DEBUG "value": [ 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG [...] 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:59 DEBUG }, 02:09:59 DEBUG "value": [ 02:09:59 DEBUG 1712970599, 02:09:59 DEBUG "1570" 02:09:59 DEBUG ] 02:09:59 DEBUG } 02:09:59 DEBUG ] 02:09:59 DEBUG } 02:09:59 DEBUG } 02:09:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:09:59 DEBUG ----- output ----- 02:09:59 DEBUG { 02:09:59 DEBUG "status": "success", 02:09:59 DEBUG "data": { 02:09:59 DEBUG "resultType": "vector", 02:09:59 DEBUG "result": [ 02:09:59 DEBUG { 02:09:59 DEBUG "metric": { 02:09:59 DEBUG "pod": "am-6d96779b5d-zn69c" 02:09:59 DEBUG }, 02:09:59 DEBUG "value": [ 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG [...] 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG 02:09:59 DEBUG "pod": "am-6d96779b5d-pfljh" 02:09:59 DEBUG }, 02:09:59 DEBUG "value": [ 02:09:59 DEBUG 1712970599, 02:09:59 DEBUG "1570.7333333333333" 02:09:59 DEBUG ] 02:09:59 DEBUG } 02:09:59 DEBUG ] 02:09:59 DEBUG } 02:09:59 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "instance": "10.142.0.107:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "instance": "10.142.0.77:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "545797729280" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "1570.4333333333334" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "282845" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-cts-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-idrepo-0" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-cts-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "29.733333333333334" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970600" 02:10:00 ERROR [http_cmd]: ERROR 02:10:00 ERROR --- status code --- 02:10:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:10:00 ERROR --- http response --- 02:10:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:10:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-idrepo-0" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "instance": "10.142.0.107:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "instance": "10.142.0.77:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "46417301.7" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "instance": "10.142.0.107:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "instance": "10.142.0.77:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "46413802.06666666" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-cts-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0.00020231666666666666" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "1570.7666666666667" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "307839392.11232704" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "5761858.902883887" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-idrepo-0" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "ds-idrepo-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "ds-cts-1" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "4.746900762058791" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "6236536832" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "instance": "10.142.0.107:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "instance": "10.142.0.77:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0.18196666666666866" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "10.507095705141445" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "instance": "10.142.0.107:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "instance": "10.142.0.77:9100" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0.06533333333333384" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970600" 02:10:00 INFO [http_cmd]: http status code OK 02:10:00 DEBUG --- status code --- 02:10:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:10:00 DEBUG --- http response --- 02:10:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:10:00 DEBUG ----- output ----- 02:10:00 DEBUG { 02:10:00 DEBUG "status": "success", 02:10:00 DEBUG "data": { 02:10:00 DEBUG "resultType": "vector", 02:10:00 DEBUG "result": [ 02:10:00 DEBUG { 02:10:00 DEBUG "metric": { 02:10:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG [...] 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG 02:10:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:10:00 DEBUG }, 02:10:00 DEBUG "value": [ 02:10:00 DEBUG 1712970600, 02:10:00 DEBUG "0" 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG ] 02:10:00 DEBUG } 02:10:00 DEBUG } 02:10:11 INFO 02:10:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:10:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:10:11 INFO [loop_until]: OK (rc = 0) 02:10:11 DEBUG --- stdout --- 02:10:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10109m 8527Mi am-6d96779b5d-qbg7l 8723m 8461Mi am-6d96779b5d-zn69c 10377m 8410Mi ds-cts-0 9889m 13450Mi ds-cts-1 10061m 13453Mi ds-cts-2 9585m 13451Mi ds-idrepo-0 7527m 5593Mi ds-idrepo-1 3702m 5522Mi ds-idrepo-2 3675m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4845m 4064Mi 02:10:11 DEBUG --- stderr --- 02:10:11 DEBUG 02:10:16 INFO 02:10:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:10:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:10:16 INFO [loop_until]: OK (rc = 0) 02:10:16 DEBUG --- stdout --- 02:10:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4721Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1849Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9086m 30% 10330Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12313m 41% 3074Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10341m 34% 10203Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4327Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11046m 36% 10373Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3874m 24% 6287Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9470m 59% 14304Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10297m 64% 14274Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3751m 23% 6682Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9949m 62% 14292Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7634m 48% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4834m 30% 7802Mi 13% 02:10:16 DEBUG --- stderr --- 02:10:16 DEBUG 02:10:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970659" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0.021" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "1575.4333333333334" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "1590.2666666666667" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "instance": "10.142.0.107:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "instance": "10.142.0.77:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "548389493760" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-cts-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "29.53333333333333" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "1590.7333333333333" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG { 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-0" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG { 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:11:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG }, 02:11:00 DEBUG { 02:11:00 DEBUG "value": [ 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG 02:11:00 DEBUG "data": { 02:11:00 DEBUG 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG 02:11:00 DEBUG "result": [ 02:11:00 DEBUG [...] 02:11:00 DEBUG { 02:11:00 DEBUG 02:11:00 DEBUG "metric": { 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG 02:11:00 DEBUG }, 02:11:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:00 DEBUG "value": [ 02:11:00 DEBUG }, 02:11:00 DEBUG 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG 02:11:00 DEBUG "282880.5" 02:11:00 DEBUG [...] 02:11:00 DEBUG ] 02:11:00 DEBUG 02:11:00 DEBUG } 02:11:00 DEBUG 02:11:00 DEBUG ] 02:11:00 DEBUG 02:11:00 DEBUG } 02:11:00 DEBUG "pod": "ds-cts-1" 02:11:00 DEBUG } 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970660" 02:11:00 ERROR [http_cmd]: ERROR 02:11:00 ERROR --- status code --- 02:11:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:11:00 ERROR --- http response --- 02:11:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:11:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-cts-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-0" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "instance": "10.142.0.107:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "instance": "10.142.0.77:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "46663568.6" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "instance": "10.142.0.107:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "instance": "10.142.0.77:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "46661287.43333334" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "1589.7" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "172929219.04761904" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "2812.598895357516" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-0" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-cts-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "4.815243986697066" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "6236893184" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "instance": "10.142.0.107:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "instance": "10.142.0.77:9100" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0.17779999999999774" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: http status code OK 02:11:00 DEBUG --- status code --- 02:11:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:00 DEBUG --- http response --- 02:11:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:11:00 DEBUG ----- output ----- 02:11:00 DEBUG { 02:11:00 DEBUG "status": "success", 02:11:00 DEBUG "data": { 02:11:00 DEBUG "resultType": "vector", 02:11:00 DEBUG "result": [ 02:11:00 DEBUG { 02:11:00 DEBUG "metric": { 02:11:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG [...] 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG 02:11:00 DEBUG "pod": "ds-idrepo-1" 02:11:00 DEBUG }, 02:11:00 DEBUG "value": [ 02:11:00 DEBUG 1712970660, 02:11:00 DEBUG "0" 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG ] 02:11:00 DEBUG } 02:11:00 DEBUG } 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970660" 02:11:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970660" 02:11:01 INFO [http_cmd]: http status code OK 02:11:01 DEBUG --- status code --- 02:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:01 DEBUG --- http response --- 02:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:11:01 DEBUG ----- output ----- 02:11:01 DEBUG { 02:11:01 DEBUG "status": "success", 02:11:01 DEBUG "data": { 02:11:01 DEBUG "resultType": "vector", 02:11:01 DEBUG "result": [ 02:11:01 DEBUG { 02:11:01 DEBUG "metric": { 02:11:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:11:01 DEBUG }, 02:11:01 DEBUG "value": [ 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG [...] 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG "pod": "ds-idrepo-1" 02:11:01 DEBUG }, 02:11:01 DEBUG "value": [ 02:11:01 DEBUG 1712970660, 02:11:01 DEBUG "3.853330920346434" 02:11:01 DEBUG ] 02:11:01 DEBUG } 02:11:01 DEBUG ] 02:11:01 DEBUG } 02:11:01 DEBUG } 02:11:01 INFO [http_cmd]: http status code OK 02:11:01 DEBUG --- status code --- 02:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:01 DEBUG --- http response --- 02:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:11:01 DEBUG ----- output ----- 02:11:01 DEBUG { 02:11:01 DEBUG "status": "success", 02:11:01 DEBUG "data": { 02:11:01 DEBUG "resultType": "vector", 02:11:01 DEBUG "result": [ 02:11:01 DEBUG { 02:11:01 DEBUG "metric": { 02:11:01 DEBUG "instance": "10.142.0.107:9100" 02:11:01 DEBUG }, 02:11:01 DEBUG "value": [ 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG [...] 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG "instance": "10.142.0.77:9100" 02:11:01 DEBUG }, 02:11:01 DEBUG "value": [ 02:11:01 DEBUG 1712970660, 02:11:01 DEBUG "0.05833333333333381" 02:11:01 DEBUG ] 02:11:01 DEBUG } 02:11:01 DEBUG ] 02:11:01 DEBUG } 02:11:01 DEBUG } 02:11:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970661" 02:11:01 INFO [http_cmd]: http status code OK 02:11:01 DEBUG --- status code --- 02:11:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:11:01 DEBUG --- http response --- 02:11:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:11:01 DEBUG ----- output ----- 02:11:01 DEBUG { 02:11:01 DEBUG "status": "success", 02:11:01 DEBUG "data": { 02:11:01 DEBUG "resultType": "vector", 02:11:01 DEBUG "result": [ 02:11:01 DEBUG { 02:11:01 DEBUG "metric": { 02:11:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:11:01 DEBUG }, 02:11:01 DEBUG "value": [ 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG [...] 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG 02:11:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:11:01 DEBUG }, 02:11:01 DEBUG "value": [ 02:11:01 DEBUG 1712970661, 02:11:01 DEBUG "0" 02:11:01 DEBUG ] 02:11:01 DEBUG } 02:11:01 DEBUG ] 02:11:01 DEBUG } 02:11:01 DEBUG } 02:11:11 INFO 02:11:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:11:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:11:11 INFO [loop_until]: OK (rc = 0) 02:11:11 DEBUG --- stdout --- 02:11:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10618m 8528Mi am-6d96779b5d-qbg7l 8632m 8460Mi am-6d96779b5d-zn69c 10393m 8408Mi ds-cts-0 9560m 13451Mi ds-cts-1 10206m 13455Mi ds-cts-2 9708m 13451Mi ds-idrepo-0 7578m 5593Mi ds-idrepo-1 3829m 5522Mi ds-idrepo-2 3717m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4615m 4064Mi 02:11:11 DEBUG --- stderr --- 02:11:11 DEBUG 02:11:16 INFO 02:11:16 INFO [loop_until]: kubectl --namespace=xlou top node 02:11:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:11:17 INFO [loop_until]: OK (rc = 0) 02:11:17 DEBUG --- stdout --- 02:11:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 1873Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9123m 30% 10326Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12369m 41% 3082Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10608m 35% 10211Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4295Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11022m 36% 10366Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3880m 24% 6286Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9719m 61% 14311Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10233m 64% 14376Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3753m 23% 6679Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9935m 62% 14289Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7694m 48% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4834m 30% 7802Mi 13% 02:11:17 DEBUG --- stderr --- 02:11:17 DEBUG 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0.021" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "1561.1333333333332" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "1564.2333333333333" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970720" 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "instance": "10.142.0.107:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "instance": "10.142.0.77:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "550417816576" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-cts-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "29.2" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "282879" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-idrepo-0" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "1560.8" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-cts-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970720" 02:12:00 ERROR [http_cmd]: ERROR 02:12:00 ERROR --- status code --- 02:12:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:12:00 ERROR --- http response --- 02:12:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:12:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-cts-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0.000022608333333333335" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-idrepo-0" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "instance": "10.142.0.107:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "instance": "10.142.0.77:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "46302771.8" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "instance": "10.142.0.107:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "instance": "10.142.0.77:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "46299989.13333333" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "1563.6666666666667" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "245118698.56669134" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "3515876.455918222" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-idrepo-0" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "ds-idrepo-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "ds-cts-1" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "4.716114090450968" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "6237798400" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "instance": "10.142.0.107:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "instance": "10.142.0.77:9100" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0.04636666666666495" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970720" 02:12:00 INFO [http_cmd]: http status code OK 02:12:00 DEBUG --- status code --- 02:12:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:00 DEBUG --- http response --- 02:12:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:00 DEBUG ----- output ----- 02:12:00 DEBUG { 02:12:00 DEBUG "status": "success", 02:12:00 DEBUG "data": { 02:12:00 DEBUG "resultType": "vector", 02:12:00 DEBUG "result": [ 02:12:00 DEBUG { 02:12:00 DEBUG "metric": { 02:12:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG [...] 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG 02:12:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:00 DEBUG }, 02:12:00 DEBUG "value": [ 02:12:00 DEBUG 1712970720, 02:12:00 DEBUG "0" 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG ] 02:12:00 DEBUG } 02:12:00 DEBUG } 02:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970721" 02:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970721" 02:12:01 INFO [http_cmd]: http status code OK 02:12:01 DEBUG --- status code --- 02:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:01 DEBUG --- http response --- 02:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:12:01 DEBUG ----- output ----- 02:12:01 DEBUG { 02:12:01 DEBUG "status": "success", 02:12:01 DEBUG "data": { 02:12:01 DEBUG "resultType": "vector", 02:12:01 DEBUG "result": [ 02:12:01 DEBUG { 02:12:01 DEBUG "metric": { 02:12:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:12:01 DEBUG }, 02:12:01 DEBUG "value": [ 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG [...] 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:01 DEBUG }, 02:12:01 DEBUG "value": [ 02:12:01 DEBUG 1712970721, 02:12:01 DEBUG "0" 02:12:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970721" 02:12:01 DEBUG ] 02:12:01 DEBUG } 02:12:01 DEBUG ] 02:12:01 DEBUG } 02:12:01 DEBUG } 02:12:01 INFO [http_cmd]: http status code OK 02:12:01 DEBUG --- status code --- 02:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:01 DEBUG --- http response --- 02:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:12:01 DEBUG ----- output ----- 02:12:01 DEBUG { 02:12:01 DEBUG "status": "success", 02:12:01 DEBUG "data": { 02:12:01 DEBUG "resultType": "vector", 02:12:01 DEBUG "result": [ 02:12:01 DEBUG { 02:12:01 DEBUG "metric": { 02:12:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:12:01 DEBUG }, 02:12:01 DEBUG "value": [ 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG [...] 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:12:01 DEBUG }, 02:12:01 DEBUG "value": [ 02:12:01 DEBUG 1712970721, 02:12:01 DEBUG "10.545383238784465" 02:12:01 DEBUG ] 02:12:01 DEBUG } 02:12:01 DEBUG ] 02:12:01 DEBUG } 02:12:01 DEBUG } 02:12:01 INFO [http_cmd]: http status code OK 02:12:01 DEBUG --- status code --- 02:12:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:12:01 DEBUG --- http response --- 02:12:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:12:01 DEBUG ----- output ----- 02:12:01 DEBUG { 02:12:01 DEBUG "status": "success", 02:12:01 DEBUG "data": { 02:12:01 DEBUG "resultType": "vector", 02:12:01 DEBUG "result": [ 02:12:01 DEBUG { 02:12:01 DEBUG "metric": { 02:12:01 DEBUG "instance": "10.142.0.107:9100" 02:12:01 DEBUG }, 02:12:01 DEBUG "value": [ 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG [...] 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG 02:12:01 DEBUG "instance": "10.142.0.77:9100" 02:12:01 DEBUG }, 02:12:01 DEBUG "value": [ 02:12:01 DEBUG 1712970721, 02:12:01 DEBUG "0.021333333333332635" 02:12:01 DEBUG ] 02:12:01 DEBUG } 02:12:01 DEBUG ] 02:12:01 DEBUG } 02:12:01 DEBUG } 02:12:11 INFO 02:12:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:12:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:12:11 INFO [loop_until]: OK (rc = 0) 02:12:11 DEBUG --- stdout --- 02:12:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10287m 8528Mi am-6d96779b5d-qbg7l 8773m 8461Mi am-6d96779b5d-zn69c 10140m 8406Mi ds-cts-0 9692m 13451Mi ds-cts-1 9852m 13453Mi ds-cts-2 9292m 13497Mi ds-idrepo-0 7442m 5593Mi ds-idrepo-1 3937m 5522Mi ds-idrepo-2 3643m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 7m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4610m 4064Mi 02:12:11 DEBUG --- stderr --- 02:12:11 DEBUG 02:12:17 INFO 02:12:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:12:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:12:17 INFO [loop_until]: OK (rc = 0) 02:12:17 DEBUG --- stdout --- 02:12:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4734Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 58m 0% 1871Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9020m 30% 10332Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11952m 40% 3080Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10443m 34% 10213Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4309Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10728m 35% 10370Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3757m 23% 6289Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9707m 61% 14304Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 9787m 61% 14358Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3764m 23% 6677Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9925m 62% 14415Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7655m 48% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4832m 30% 7804Mi 13% 02:12:17 DEBUG --- stderr --- 02:12:17 DEBUG 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0.021" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "1571.4333333333334" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "1571" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "instance": "10.142.0.107:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "instance": "10.142.0.77:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "552786512896" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-cts-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "29.233333333333334" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "282832" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-idrepo-0" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "1571.2333333333333" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-cts-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970780" 02:13:00 ERROR [http_cmd]: ERROR 02:13:00 ERROR --- status code --- 02:13:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:13:00 ERROR --- http response --- 02:13:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:13:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "instance": "10.142.0.107:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "instance": "10.142.0.77:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "46402882.46666667" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-cts-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0.00046666666666666666" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-idrepo-0" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "instance": "10.142.0.107:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "instance": "10.142.0.77:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "46399401.1" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "1571.9666666666667" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "820.1568496579343" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG { 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-idrepo-0" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "546.7712331052895" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "ds-idrepo-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "ds-cts-1" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "4.734611594656939" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "6237253632" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "instance": "10.142.0.107:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "instance": "10.142.0.77:9100" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0.14836666666666407" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970780" 02:13:00 INFO [http_cmd]: http status code OK 02:13:00 DEBUG --- status code --- 02:13:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:00 DEBUG --- http response --- 02:13:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:00 DEBUG ----- output ----- 02:13:00 DEBUG { 02:13:00 DEBUG "status": "success", 02:13:00 DEBUG "data": { 02:13:00 DEBUG "resultType": "vector", 02:13:00 DEBUG "result": [ 02:13:00 DEBUG { 02:13:00 DEBUG "metric": { 02:13:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG [...] 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG 02:13:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:13:00 DEBUG }, 02:13:00 DEBUG "value": [ 02:13:00 DEBUG 1712970780, 02:13:00 DEBUG "0" 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG ] 02:13:00 DEBUG } 02:13:00 DEBUG } 02:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970781" 02:13:01 INFO [http_cmd]: http status code OK 02:13:01 DEBUG --- status code --- 02:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:01 DEBUG --- http response --- 02:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:13:01 DEBUG ----- output ----- 02:13:01 DEBUG { 02:13:01 DEBUG "status": "success", 02:13:01 DEBUG "data": { 02:13:01 DEBUG "resultType": "vector", 02:13:01 DEBUG "result": [ 02:13:01 DEBUG { 02:13:01 DEBUG "metric": { 02:13:01 DEBUG "instance": "10.142.0.107:9100" 02:13:01 DEBUG }, 02:13:01 DEBUG "value": [ 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG [...] 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG "instance": "10.142.0.77:9100" 02:13:01 DEBUG }, 02:13:01 DEBUG "value": [ 02:13:01 DEBUG 1712970781, 02:13:01 DEBUG "0.04933333333333465" 02:13:01 DEBUG ] 02:13:01 DEBUG } 02:13:01 DEBUG ] 02:13:01 DEBUG } 02:13:01 DEBUG } 02:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970781" 02:13:01 INFO [http_cmd]: http status code OK 02:13:01 DEBUG --- status code --- 02:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:01 DEBUG --- http response --- 02:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:13:01 DEBUG ----- output ----- 02:13:01 DEBUG { 02:13:01 DEBUG "status": "success", 02:13:01 DEBUG "data": { 02:13:01 DEBUG "resultType": "vector", 02:13:01 DEBUG "result": [ 02:13:01 DEBUG { 02:13:01 DEBUG "metric": { 02:13:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:13:01 DEBUG }, 02:13:01 DEBUG "value": [ 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG [...] 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:13:01 DEBUG }, 02:13:01 DEBUG "value": [ 02:13:01 DEBUG 1712970781, 02:13:01 DEBUG "0" 02:13:01 DEBUG ] 02:13:01 DEBUG } 02:13:01 DEBUG ] 02:13:01 DEBUG } 02:13:01 DEBUG } 02:13:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970781" 02:13:01 INFO [http_cmd]: http status code OK 02:13:01 DEBUG --- status code --- 02:13:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:13:01 DEBUG --- http response --- 02:13:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:13:01 DEBUG ----- output ----- 02:13:01 DEBUG { 02:13:01 DEBUG "status": "success", 02:13:01 DEBUG "data": { 02:13:01 DEBUG "resultType": "vector", 02:13:01 DEBUG "result": [ 02:13:01 DEBUG { 02:13:01 DEBUG "metric": { 02:13:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:13:01 DEBUG }, 02:13:01 DEBUG "value": [ 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG [...] 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG 02:13:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:13:01 DEBUG }, 02:13:01 DEBUG "value": [ 02:13:01 DEBUG 1712970781, 02:13:01 DEBUG "0.0031254463540802604" 02:13:01 DEBUG ] 02:13:01 DEBUG } 02:13:01 DEBUG ] 02:13:01 DEBUG } 02:13:01 DEBUG } 02:13:11 INFO 02:13:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:13:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:13:11 INFO [loop_until]: OK (rc = 0) 02:13:11 DEBUG --- stdout --- 02:13:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10352m 8532Mi am-6d96779b5d-qbg7l 8644m 8463Mi am-6d96779b5d-zn69c 9966m 8407Mi ds-cts-0 9554m 13449Mi ds-cts-1 9671m 13453Mi ds-cts-2 9250m 13450Mi ds-idrepo-0 7652m 5593Mi ds-idrepo-1 3808m 5522Mi ds-idrepo-2 3694m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4594m 4064Mi 02:13:11 DEBUG --- stderr --- 02:13:11 DEBUG 02:13:17 INFO 02:13:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:13:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:13:17 INFO [loop_until]: OK (rc = 0) 02:13:17 DEBUG --- stdout --- 02:13:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4736Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 1864Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8946m 29% 10334Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12272m 41% 3071Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10479m 35% 10213Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 134m 0% 4298Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11002m 36% 10371Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3853m 24% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9549m 60% 14309Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 9657m 60% 14266Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3748m 23% 6681Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9749m 61% 14306Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7650m 48% 6384Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4824m 30% 7802Mi 13% 02:13:17 DEBUG --- stderr --- 02:13:17 DEBUG 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0.021" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "1565.6" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "1589.6333333333332" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "instance": "10.142.0.107:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "instance": "10.142.0.77:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "555007800320" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-idrepo-0" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "282727" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-cts-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "27.8" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "1565.5333333333333" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 ERROR [http_cmd]: ERROR 02:14:00 DEBUG --- status code --- 02:14:00 ERROR --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 ERROR --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-cts-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:14:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-cts-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0.00048318333333333336" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "instance": "10.142.0.107:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "instance": "10.142.0.77:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "46641401.7" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-idrepo-0" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "instance": "10.142.0.107:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "instance": "10.142.0.77:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "46638178.36666666" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "1565.0333333333333" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "2387547.936422232" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "4284.95258664327" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-idrepo-0" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "ds-idrepo-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "ds-cts-1" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "4.706104353836902" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "6238121984" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "instance": "10.142.0.107:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "instance": "10.142.0.77:9100" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0.1370333333333316" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970840" 02:14:00 INFO [http_cmd]: http status code OK 02:14:00 DEBUG --- status code --- 02:14:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:00 DEBUG --- http response --- 02:14:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:00 DEBUG ----- output ----- 02:14:00 DEBUG { 02:14:00 DEBUG "status": "success", 02:14:00 DEBUG "data": { 02:14:00 DEBUG "resultType": "vector", 02:14:00 DEBUG "result": [ 02:14:00 DEBUG { 02:14:00 DEBUG "metric": { 02:14:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG [...] 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG 02:14:00 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:14:00 DEBUG }, 02:14:00 DEBUG "value": [ 02:14:00 DEBUG 1712970840, 02:14:00 DEBUG "0" 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG ] 02:14:00 DEBUG } 02:14:00 DEBUG } 02:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970841" 02:14:01 INFO [http_cmd]: http status code OK 02:14:01 DEBUG --- status code --- 02:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:01 DEBUG --- http response --- 02:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:14:01 DEBUG ----- output ----- 02:14:01 DEBUG { 02:14:01 DEBUG "status": "success", 02:14:01 DEBUG "data": { 02:14:01 DEBUG "resultType": "vector", 02:14:01 DEBUG "result": [ 02:14:01 DEBUG { 02:14:01 DEBUG "metric": { 02:14:01 DEBUG "instance": "10.142.0.107:9100" 02:14:01 DEBUG }, 02:14:01 DEBUG "value": [ 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG [...] 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG "instance": "10.142.0.77:9100" 02:14:01 DEBUG }, 02:14:01 DEBUG "value": [ 02:14:01 DEBUG 1712970841, 02:14:01 DEBUG "0.04933333333333228" 02:14:01 DEBUG ] 02:14:01 DEBUG } 02:14:01 DEBUG ] 02:14:01 DEBUG } 02:14:01 DEBUG } 02:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970841" 02:14:01 INFO [http_cmd]: http status code OK 02:14:01 DEBUG --- status code --- 02:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:01 DEBUG --- http response --- 02:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:14:01 DEBUG ----- output ----- 02:14:01 DEBUG { 02:14:01 DEBUG "status": "success", 02:14:01 DEBUG "data": { 02:14:01 DEBUG "resultType": "vector", 02:14:01 DEBUG "result": [ 02:14:01 DEBUG { 02:14:01 DEBUG "metric": { 02:14:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:14:01 DEBUG }, 02:14:01 DEBUG "value": [ 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG [...] 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:14:01 DEBUG }, 02:14:01 DEBUG "value": [ 02:14:01 DEBUG 1712970841, 02:14:01 DEBUG "0" 02:14:01 DEBUG ] 02:14:01 DEBUG } 02:14:01 DEBUG ] 02:14:01 DEBUG } 02:14:01 DEBUG } 02:14:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970841" 02:14:01 INFO [http_cmd]: http status code OK 02:14:01 DEBUG --- status code --- 02:14:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:14:01 DEBUG --- http response --- 02:14:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:14:01 DEBUG ----- output ----- 02:14:01 DEBUG { 02:14:01 DEBUG "status": "success", 02:14:01 DEBUG "data": { 02:14:01 DEBUG "resultType": "vector", 02:14:01 DEBUG "result": [ 02:14:01 DEBUG { 02:14:01 DEBUG "metric": { 02:14:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:14:01 DEBUG }, 02:14:01 DEBUG "value": [ 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG [...] 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG 02:14:01 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:14:01 DEBUG }, 02:14:01 DEBUG "value": [ 02:14:01 DEBUG 1712970841, 02:14:01 DEBUG "0.0033468093004420723" 02:14:01 DEBUG ] 02:14:01 DEBUG } 02:14:01 DEBUG ] 02:14:01 DEBUG } 02:14:01 DEBUG } 02:14:11 INFO 02:14:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:14:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:14:11 INFO [loop_until]: OK (rc = 0) 02:14:11 DEBUG --- stdout --- 02:14:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10471m 8528Mi am-6d96779b5d-qbg7l 8493m 8462Mi am-6d96779b5d-zn69c 10433m 8415Mi ds-cts-0 9686m 13451Mi ds-cts-1 9711m 13507Mi ds-cts-2 9651m 13449Mi ds-idrepo-0 7468m 5593Mi ds-idrepo-1 3762m 5522Mi ds-idrepo-2 3635m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 8m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4736m 4064Mi 02:14:11 DEBUG --- stderr --- 02:14:11 DEBUG 02:14:17 INFO 02:14:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:14:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:14:17 INFO [loop_until]: OK (rc = 0) 02:14:17 DEBUG --- stdout --- 02:14:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4733Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 58m 0% 1865Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8939m 29% 10332Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12313m 41% 3077Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10406m 34% 10212Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4325Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11114m 37% 10366Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3904m 24% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9432m 59% 14299Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10075m 63% 14276Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3757m 23% 6683Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9948m 62% 14293Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7656m 48% 6365Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4691m 29% 7803Mi 13% 02:14:17 DEBUG --- stderr --- 02:14:17 DEBUG 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0.021" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "1575.3333333333333" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-idrepo-0" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "1559.4" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "instance": "10.142.0.107:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "instance": "10.142.0.77:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "557529666560" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "282792.5" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "1575.2" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-cts-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "28.866666666666667" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970900" 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 ERROR [http_cmd]: ERROR 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 ERROR --- status code --- 02:15:00 DEBUG --- status code --- 02:15:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 ERROR --- http response --- 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:15:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-cts-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "instance": "10.142.0.107:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "instance": "10.142.0.77:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "46517190.4" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-cts-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0.00019999999999999998" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-idrepo-0" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "instance": "10.142.0.107:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "instance": "10.142.0.77:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "46514115.3" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "1575.0333333333333" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "6860345.861254088" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-idrepo-0" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "pod": "ds-idrepo-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "pod": "ds-cts-1" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "4.726224325809075" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "6238859264" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970900" 02:15:00 INFO [http_cmd]: http status code OK 02:15:00 DEBUG --- status code --- 02:15:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:00 DEBUG --- http response --- 02:15:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:00 DEBUG ----- output ----- 02:15:00 DEBUG { 02:15:00 DEBUG "status": "success", 02:15:00 DEBUG "data": { 02:15:00 DEBUG "resultType": "vector", 02:15:00 DEBUG "result": [ 02:15:00 DEBUG { 02:15:00 DEBUG "metric": { 02:15:00 DEBUG "instance": "10.142.0.107:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG [...] 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG 02:15:00 DEBUG "instance": "10.142.0.77:9100" 02:15:00 DEBUG }, 02:15:00 DEBUG "value": [ 02:15:00 DEBUG 1712970900, 02:15:00 DEBUG "0.16460000000000102" 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG ] 02:15:00 DEBUG } 02:15:00 DEBUG } 02:15:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970900" 02:15:01 INFO [http_cmd]: http status code OK 02:15:01 DEBUG --- status code --- 02:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:01 DEBUG --- http response --- 02:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:01 DEBUG ----- output ----- 02:15:01 DEBUG { 02:15:01 DEBUG "status": "success", 02:15:01 DEBUG "data": { 02:15:01 DEBUG "resultType": "vector", 02:15:01 DEBUG "result": [ 02:15:01 DEBUG { 02:15:01 DEBUG "metric": { 02:15:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG [...] 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 1712970900, 02:15:01 DEBUG "0" 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG } 02:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970901" 02:15:01 INFO [http_cmd]: http status code OK 02:15:01 DEBUG --- status code --- 02:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:01 DEBUG --- http response --- 02:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:15:01 DEBUG ----- output ----- 02:15:01 DEBUG { 02:15:01 DEBUG "status": "success", 02:15:01 DEBUG "data": { 02:15:01 DEBUG "resultType": "vector", 02:15:01 DEBUG "result": [ 02:15:01 DEBUG { 02:15:01 DEBUG "metric": { 02:15:01 DEBUG "instance": "10.142.0.107:9100" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG [...] 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG "instance": "10.142.0.77:9100" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 1712970901, 02:15:01 DEBUG "0.05133333333333069" 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG } 02:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970901" 02:15:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970901" 02:15:01 INFO [http_cmd]: http status code OK 02:15:01 DEBUG --- status code --- 02:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:01 DEBUG --- http response --- 02:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:15:01 DEBUG ----- output ----- 02:15:01 DEBUG { 02:15:01 DEBUG "status": "success", 02:15:01 DEBUG "data": { 02:15:01 DEBUG "resultType": "vector", 02:15:01 DEBUG "result": [ 02:15:01 DEBUG { 02:15:01 DEBUG "metric": { 02:15:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG [...] 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 1712970901, 02:15:01 DEBUG "0" 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG } 02:15:01 INFO [http_cmd]: http status code OK 02:15:01 DEBUG --- status code --- 02:15:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:15:01 DEBUG --- http response --- 02:15:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:15:01 DEBUG ----- output ----- 02:15:01 DEBUG { 02:15:01 DEBUG "status": "success", 02:15:01 DEBUG "data": { 02:15:01 DEBUG "resultType": "vector", 02:15:01 DEBUG "result": [ 02:15:01 DEBUG { 02:15:01 DEBUG "metric": { 02:15:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG [...] 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG 02:15:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:15:01 DEBUG }, 02:15:01 DEBUG "value": [ 02:15:01 DEBUG 1712970901, 02:15:01 DEBUG "0" 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG ] 02:15:01 DEBUG } 02:15:01 DEBUG } 02:15:11 INFO 02:15:11 INFO [loop_until]: kubectl --namespace=xlou top pods 02:15:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:15:11 INFO [loop_until]: OK (rc = 0) 02:15:11 DEBUG --- stdout --- 02:15:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10316m 8531Mi am-6d96779b5d-qbg7l 8591m 8460Mi am-6d96779b5d-zn69c 10025m 8416Mi ds-cts-0 9863m 13450Mi ds-cts-1 9862m 13535Mi ds-cts-2 9396m 13451Mi ds-idrepo-0 7571m 5593Mi ds-idrepo-1 3793m 5521Mi ds-idrepo-2 3630m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4692m 4064Mi 02:15:11 DEBUG --- stderr --- 02:15:11 DEBUG 02:15:17 INFO 02:15:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:15:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:15:17 INFO [loop_until]: OK (rc = 0) 02:15:17 DEBUG --- stdout --- 02:15:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4741Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 1864Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8933m 29% 10330Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12290m 41% 3085Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10501m 35% 10218Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4301Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10979m 36% 10373Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3928m 24% 6290Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9588m 60% 14306Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10197m 64% 14272Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3768m 23% 6681Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9603m 60% 14302Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7639m 48% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4861m 30% 7803Mi 13% 02:15:17 DEBUG --- stderr --- 02:15:17 DEBUG 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0.021" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "1578.6" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "1571.1" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-idrepo-0" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970960" 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG { 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "282776" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "instance": "10.142.0.107:9100" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "instance": "10.142.0.77:9100" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "559555034112" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:00 DEBUG }, 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-cts-1" 02:16:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:00 DEBUG }, 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "28.833333333333332" 02:16:00 DEBUG "1578.2666666666667" 02:16:00 DEBUG ] 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712970960" 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 ERROR [http_cmd]: ERROR 02:16:00 ERROR --- status code --- 02:16:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:16:00 ERROR --- http response --- 02:16:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:16:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-cts-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 DEBUG "instance": "10.142.0.107:9100" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "instance": "10.142.0.77:9100" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "46080822.36666667" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-idrepo-0" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "instance": "10.142.0.107:9100" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "instance": "10.142.0.77:9100" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "46077613.13333333" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-cts-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0.00078535" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "1578.7" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "1018.2127712805491" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-idrepo-0" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "pod": "ds-idrepo-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "pod": "ds-cts-1" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "0" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "4.786153191360324" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712970960" 02:16:00 INFO [http_cmd]: http status code OK 02:16:00 DEBUG --- status code --- 02:16:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:00 DEBUG --- http response --- 02:16:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:00 DEBUG ----- output ----- 02:16:00 DEBUG { 02:16:00 DEBUG "status": "success", 02:16:00 DEBUG "data": { 02:16:00 DEBUG "resultType": "vector", 02:16:00 DEBUG "result": [ 02:16:00 DEBUG { 02:16:00 DEBUG "metric": { 02:16:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG [...] 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG 02:16:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:16:00 DEBUG }, 02:16:00 DEBUG "value": [ 02:16:00 DEBUG 1712970960, 02:16:00 DEBUG "6235619328" 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG ] 02:16:00 DEBUG } 02:16:00 DEBUG } 02:16:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970960" 02:16:01 INFO [http_cmd]: http status code OK 02:16:01 DEBUG --- status code --- 02:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:01 DEBUG --- http response --- 02:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:01 DEBUG ----- output ----- 02:16:01 DEBUG { 02:16:01 DEBUG "status": "success", 02:16:01 DEBUG "data": { 02:16:01 DEBUG "resultType": "vector", 02:16:01 DEBUG "result": [ 02:16:01 DEBUG { 02:16:01 DEBUG "metric": { 02:16:01 DEBUG "instance": "10.142.0.107:9100" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG [...] 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG "instance": "10.142.0.77:9100" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 1712970960, 02:16:01 DEBUG "0.0466999999999994" 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG } 02:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970961" 02:16:01 INFO [http_cmd]: http status code OK 02:16:01 DEBUG --- status code --- 02:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:01 DEBUG --- http response --- 02:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:16:01 DEBUG ----- output ----- 02:16:01 DEBUG { 02:16:01 DEBUG "status": "success", 02:16:01 DEBUG "data": { 02:16:01 DEBUG "resultType": "vector", 02:16:01 DEBUG "result": [ 02:16:01 DEBUG { 02:16:01 DEBUG "metric": { 02:16:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG [...] 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 1712970961, 02:16:01 DEBUG "0" 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG } 02:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712970961" 02:16:01 INFO [http_cmd]: http status code OK 02:16:01 DEBUG --- status code --- 02:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:01 DEBUG --- http response --- 02:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:16:01 DEBUG ----- output ----- 02:16:01 DEBUG { 02:16:01 DEBUG "status": "success", 02:16:01 DEBUG "data": { 02:16:01 DEBUG "resultType": "vector", 02:16:01 DEBUG "result": [ 02:16:01 DEBUG { 02:16:01 DEBUG "metric": { 02:16:01 DEBUG "instance": "10.142.0.107:9100" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG [...] 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG "instance": "10.142.0.77:9100" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 1712970961, 02:16:01 DEBUG "0.019666666666665597" 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG } 02:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970961" 02:16:01 INFO [http_cmd]: http status code OK 02:16:01 DEBUG --- status code --- 02:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:01 DEBUG --- http response --- 02:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:16:01 DEBUG ----- output ----- 02:16:01 DEBUG { 02:16:01 DEBUG "status": "success", 02:16:01 DEBUG "data": { 02:16:01 DEBUG "resultType": "vector", 02:16:01 DEBUG "result": [ 02:16:01 DEBUG { 02:16:01 DEBUG "metric": { 02:16:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG [...] 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 1712970961, 02:16:01 DEBUG "0" 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG } 02:16:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712970961" 02:16:01 INFO [http_cmd]: http status code OK 02:16:01 DEBUG --- status code --- 02:16:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:16:01 DEBUG --- http response --- 02:16:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:16:01 DEBUG ----- output ----- 02:16:01 DEBUG { 02:16:01 DEBUG "status": "success", 02:16:01 DEBUG "data": { 02:16:01 DEBUG "resultType": "vector", 02:16:01 DEBUG "result": [ 02:16:01 DEBUG { 02:16:01 DEBUG "metric": { 02:16:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG [...] 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG 02:16:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:16:01 DEBUG }, 02:16:01 DEBUG "value": [ 02:16:01 DEBUG 1712970961, 02:16:01 DEBUG "0.007655842114088465" 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG ] 02:16:01 DEBUG } 02:16:01 DEBUG } 02:16:12 INFO 02:16:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:16:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:16:12 INFO [loop_until]: OK (rc = 0) 02:16:12 DEBUG --- stdout --- 02:16:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10680m 8529Mi am-6d96779b5d-qbg7l 8672m 8464Mi am-6d96779b5d-zn69c 10506m 8424Mi ds-cts-0 9603m 13450Mi ds-cts-1 10034m 13454Mi ds-cts-2 9394m 13451Mi ds-idrepo-0 7635m 5593Mi ds-idrepo-1 3933m 5522Mi ds-idrepo-2 3658m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4824m 4064Mi 02:16:12 DEBUG --- stderr --- 02:16:12 DEBUG 02:16:17 INFO 02:16:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:16:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:16:17 INFO [loop_until]: OK (rc = 0) 02:16:17 DEBUG --- stdout --- 02:16:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1858Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9125m 30% 10324Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11873m 39% 3083Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10542m 35% 10227Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4300Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10965m 36% 10375Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3861m 24% 6287Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9704m 61% 14297Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10175m 64% 14277Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3635m 22% 6680Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9579m 60% 14292Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7673m 48% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4824m 30% 7804Mi 13% 02:16:17 DEBUG --- stderr --- 02:16:17 DEBUG 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "0.021" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "1567.5666666666666" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG { 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "data": { 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG "result": [ 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG { 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG }, 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG "value": [ 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG [...] 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:00 DEBUG "pod": "ds-idrepo-0" 02:17:00 DEBUG }, 02:17:00 DEBUG }, 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG "value": [ 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "282587.5" 02:17:00 DEBUG "1564.9" 02:17:00 DEBUG "0" 02:17:00 DEBUG ] 02:17:00 DEBUG ] 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG ] 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971020" 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "instance": "10.142.0.107:9100" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "instance": "10.142.0.77:9100" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "561948552192" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-cts-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "29.666666666666668" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971020" 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 ERROR [http_cmd]: ERROR 02:17:00 ERROR --- status code --- 02:17:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:17:00 ERROR --- http response --- 02:17:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:17:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "1568.2" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-cts-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "0" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971020" 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-cts-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "0.0002666666666666667" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "instance": "10.142.0.107:9100" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "instance": "10.142.0.77:9100" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "46507181.233333334" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG "data": { 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG "result": [ 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-idrepo-0" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "0" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "instance": "10.142.0.107:9100" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "instance": "10.142.0.77:9100" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "46511119.766666666" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "1567.3333333333333" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "166.1569711268397" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "1661.569711268397" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-idrepo-0" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "0" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "pod": "ds-idrepo-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "pod": "ds-cts-1" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "0" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "4.728035351936271" 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG ] 02:17:00 DEBUG } 02:17:00 DEBUG } 02:17:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971020" 02:17:00 INFO [http_cmd]: http status code OK 02:17:00 DEBUG --- status code --- 02:17:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:00 DEBUG --- http response --- 02:17:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:00 DEBUG ----- output ----- 02:17:00 DEBUG { 02:17:00 DEBUG "status": "success", 02:17:00 DEBUG "data": { 02:17:00 DEBUG "resultType": "vector", 02:17:00 DEBUG "result": [ 02:17:00 DEBUG { 02:17:00 DEBUG "metric": { 02:17:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG [...] 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG 02:17:00 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:17:00 DEBUG }, 02:17:00 DEBUG "value": [ 02:17:00 DEBUG 1712971020, 02:17:00 DEBUG "6236393472" 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG } 02:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971021" 02:17:01 INFO [http_cmd]: http status code OK 02:17:01 DEBUG --- status code --- 02:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:01 DEBUG --- http response --- 02:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:01 DEBUG ----- output ----- 02:17:01 DEBUG { 02:17:01 DEBUG "status": "success", 02:17:01 DEBUG "data": { 02:17:01 DEBUG "resultType": "vector", 02:17:01 DEBUG "result": [ 02:17:01 DEBUG { 02:17:01 DEBUG "metric": { 02:17:01 DEBUG "instance": "10.142.0.107:9100" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG [...] 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG "instance": "10.142.0.77:9100" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 1712971021, 02:17:01 DEBUG "0.17136666666667585" 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG } 02:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971021" 02:17:01 INFO [http_cmd]: http status code OK 02:17:01 DEBUG --- status code --- 02:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:01 DEBUG --- http response --- 02:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:01 DEBUG ----- output ----- 02:17:01 DEBUG { 02:17:01 DEBUG "status": "success", 02:17:01 DEBUG "data": { 02:17:01 DEBUG "resultType": "vector", 02:17:01 DEBUG "result": [ 02:17:01 DEBUG { 02:17:01 DEBUG "metric": { 02:17:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG [...] 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG "pod": "ds-idrepo-1" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 1712971021, 02:17:01 DEBUG "0" 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG } 02:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971021" 02:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971021" 02:17:01 INFO [http_cmd]: http status code OK 02:17:01 DEBUG --- status code --- 02:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:01 DEBUG --- http response --- 02:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:17:01 DEBUG ----- output ----- 02:17:01 DEBUG { 02:17:01 DEBUG "status": "success", 02:17:01 DEBUG "data": { 02:17:01 DEBUG "resultType": "vector", 02:17:01 DEBUG "result": [ 02:17:01 DEBUG { 02:17:01 DEBUG "metric": { 02:17:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG [...] 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG "pod": "ds-idrepo-1" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 1712971021, 02:17:01 DEBUG "2.226196015800174" 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG } 02:17:01 INFO [http_cmd]: http status code OK 02:17:01 DEBUG --- status code --- 02:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:01 DEBUG --- http response --- 02:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:17:01 DEBUG ----- output ----- 02:17:01 DEBUG { 02:17:01 DEBUG "status": "success", 02:17:01 DEBUG "data": { 02:17:01 DEBUG "resultType": "vector", 02:17:01 DEBUG "result": [ 02:17:01 DEBUG { 02:17:01 DEBUG "metric": { 02:17:01 DEBUG "instance": "10.142.0.107:9100" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG [...] 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG "instance": "10.142.0.77:9100" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 1712971021, 02:17:01 DEBUG "0.05333333333333172" 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG } 02:17:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971021" 02:17:01 INFO [http_cmd]: http status code OK 02:17:01 DEBUG --- status code --- 02:17:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:17:01 DEBUG --- http response --- 02:17:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:17:01 DEBUG ----- output ----- 02:17:01 DEBUG { 02:17:01 DEBUG "status": "success", 02:17:01 DEBUG "data": { 02:17:01 DEBUG "resultType": "vector", 02:17:01 DEBUG "result": [ 02:17:01 DEBUG { 02:17:01 DEBUG "metric": { 02:17:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG [...] 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG 02:17:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:17:01 DEBUG }, 02:17:01 DEBUG "value": [ 02:17:01 DEBUG 1712971021, 02:17:01 DEBUG "0" 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG ] 02:17:01 DEBUG } 02:17:01 DEBUG } 02:17:12 INFO 02:17:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:17:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:17:12 INFO [loop_until]: OK (rc = 0) 02:17:12 DEBUG --- stdout --- 02:17:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10267m 8534Mi am-6d96779b5d-qbg7l 8982m 8465Mi am-6d96779b5d-zn69c 9830m 8422Mi ds-cts-0 9740m 13450Mi ds-cts-1 10114m 13452Mi ds-cts-2 9715m 13451Mi ds-idrepo-0 7697m 5593Mi ds-idrepo-1 3730m 5522Mi ds-idrepo-2 3569m 5860Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4683m 4064Mi 02:17:12 DEBUG --- stderr --- 02:17:12 DEBUG 02:17:17 INFO 02:17:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:17:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:17:17 INFO [loop_until]: OK (rc = 0) 02:17:17 DEBUG --- stdout --- 02:17:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4735Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1860Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8903m 29% 10330Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12328m 41% 3089Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10463m 35% 10220Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4334Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11103m 37% 10366Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3904m 24% 6292Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9741m 61% 14308Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10226m 64% 14393Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3731m 23% 6681Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9867m 62% 14306Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7654m 48% 6367Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4844m 30% 7804Mi 13% 02:17:17 DEBUG --- stderr --- 02:17:17 DEBUG 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0.021" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "1573.2333333333333" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "1574.9666666666667" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG }, 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971080" 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:00 DEBUG 02:18:00 DEBUG "pod": "ds-idrepo-0" 02:18:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:00 DEBUG }, 02:18:00 DEBUG "pod": "ds-cts-1" 02:18:00 DEBUG }, 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "282504" 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0" 02:18:00 DEBUG "1573.2666666666667" 02:18:00 DEBUG ] 02:18:00 DEBUG "29.5" 02:18:00 DEBUG ] 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "instance": "10.142.0.107:9100" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "instance": "10.142.0.77:9100" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "564523926528" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971080" 02:18:00 ERROR [http_cmd]: ERROR 02:18:00 ERROR --- status code --- 02:18:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:18:00 ERROR --- http response --- 02:18:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:18:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "ds-cts-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "ds-cts-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "1573.6666666666667" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:00 DEBUG ] 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG } 02:18:00 DEBUG { 02:18:00 DEBUG } 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "instance": "10.142.0.107:9100" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "instance": "10.142.0.77:9100" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "46484458.93333334" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG { 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "ds-idrepo-0" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "instance": "10.142.0.107:9100" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "instance": "10.142.0.77:9100" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "46487404.36666666" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "297570884.5647995" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "6465638.147910834" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "ds-idrepo-0" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971080" 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "4.725104969206842" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:00 INFO [http_cmd]: http status code OK 02:18:00 DEBUG --- status code --- 02:18:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:00 DEBUG --- http response --- 02:18:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:18:00 DEBUG ----- output ----- 02:18:00 DEBUG { 02:18:00 DEBUG "status": "success", 02:18:00 DEBUG "data": { 02:18:00 DEBUG "resultType": "vector", 02:18:00 DEBUG "result": [ 02:18:00 DEBUG { 02:18:00 DEBUG "metric": { 02:18:00 DEBUG "pod": "ds-idrepo-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG [...] 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG 02:18:00 DEBUG "pod": "ds-cts-1" 02:18:00 DEBUG }, 02:18:00 DEBUG "value": [ 02:18:00 DEBUG 1712971080, 02:18:00 DEBUG "0" 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG ] 02:18:00 DEBUG } 02:18:00 DEBUG } 02:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971081" 02:18:01 INFO [http_cmd]: http status code OK 02:18:01 DEBUG --- status code --- 02:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:01 DEBUG --- http response --- 02:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:01 DEBUG ----- output ----- 02:18:01 DEBUG { 02:18:01 DEBUG "status": "success", 02:18:01 DEBUG "data": { 02:18:01 DEBUG "resultType": "vector", 02:18:01 DEBUG "result": [ 02:18:01 DEBUG { 02:18:01 DEBUG "metric": { 02:18:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG [...] 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 1712971081, 02:18:01 DEBUG "6236303360" 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG } 02:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971081" 02:18:01 INFO [http_cmd]: http status code OK 02:18:01 DEBUG --- status code --- 02:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:01 DEBUG --- http response --- 02:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:01 DEBUG ----- output ----- 02:18:01 DEBUG { 02:18:01 DEBUG "status": "success", 02:18:01 DEBUG "data": { 02:18:01 DEBUG "resultType": "vector", 02:18:01 DEBUG "result": [ 02:18:01 DEBUG { 02:18:01 DEBUG "metric": { 02:18:01 DEBUG "instance": "10.142.0.107:9100" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG [...] 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG "instance": "10.142.0.77:9100" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 1712971081, 02:18:01 DEBUG "0.16920000000000357" 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG } 02:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971081" 02:18:01 INFO [http_cmd]: http status code OK 02:18:01 DEBUG --- status code --- 02:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:01 DEBUG --- http response --- 02:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:18:01 DEBUG ----- output ----- 02:18:01 DEBUG { 02:18:01 DEBUG "status": "success", 02:18:01 DEBUG "data": { 02:18:01 DEBUG "resultType": "vector", 02:18:01 DEBUG "result": [ 02:18:01 DEBUG { 02:18:01 DEBUG "metric": { 02:18:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG [...] 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 1712971081, 02:18:01 DEBUG "0" 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG } 02:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971081" 02:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971081" 02:18:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971081" 02:18:01 INFO [http_cmd]: http status code OK 02:18:01 DEBUG --- status code --- 02:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:01 DEBUG --- http response --- 02:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:18:01 DEBUG ----- output ----- 02:18:01 DEBUG { 02:18:01 DEBUG "status": "success", 02:18:01 DEBUG "data": { 02:18:01 DEBUG "resultType": "vector", 02:18:01 DEBUG "result": [ 02:18:01 DEBUG { 02:18:01 DEBUG "metric": { 02:18:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG [...] 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 1712971081, 02:18:01 DEBUG "0" 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG } 02:18:01 INFO [http_cmd]: http status code OK 02:18:01 DEBUG --- status code --- 02:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:01 DEBUG --- http response --- 02:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:18:01 DEBUG ----- output ----- 02:18:01 DEBUG { 02:18:01 DEBUG "status": "success", 02:18:01 DEBUG "data": { 02:18:01 DEBUG "resultType": "vector", 02:18:01 DEBUG "result": [ 02:18:01 DEBUG { 02:18:01 DEBUG "metric": { 02:18:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG [...] 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 1712971081, 02:18:01 DEBUG "0.0015037019684293789" 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG } 02:18:01 INFO [http_cmd]: http status code OK 02:18:01 DEBUG --- status code --- 02:18:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:18:01 DEBUG --- http response --- 02:18:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:18:01 DEBUG ----- output ----- 02:18:01 DEBUG { 02:18:01 DEBUG "status": "success", 02:18:01 DEBUG "data": { 02:18:01 DEBUG "resultType": "vector", 02:18:01 DEBUG "result": [ 02:18:01 DEBUG { 02:18:01 DEBUG "metric": { 02:18:01 DEBUG "instance": "10.142.0.107:9100" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG [...] 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG 02:18:01 DEBUG "instance": "10.142.0.77:9100" 02:18:01 DEBUG }, 02:18:01 DEBUG "value": [ 02:18:01 DEBUG 1712971081, 02:18:01 DEBUG "0.06766666666666718" 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG ] 02:18:01 DEBUG } 02:18:01 DEBUG } 02:18:12 INFO 02:18:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:18:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:18:12 INFO [loop_until]: OK (rc = 0) 02:18:12 DEBUG --- stdout --- 02:18:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10603m 8532Mi am-6d96779b5d-qbg7l 8698m 8462Mi am-6d96779b5d-zn69c 9964m 8412Mi ds-cts-0 9561m 13449Mi ds-cts-1 9867m 13554Mi ds-cts-2 9834m 13450Mi ds-idrepo-0 7558m 5593Mi ds-idrepo-1 3747m 5522Mi ds-idrepo-2 3652m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4737m 4064Mi 02:18:12 DEBUG --- stderr --- 02:18:12 DEBUG 02:18:17 INFO 02:18:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:18:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:18:17 INFO [loop_until]: OK (rc = 0) 02:18:17 DEBUG --- stdout --- 02:18:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4734Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 1859Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8702m 29% 10342Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12358m 41% 3095Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10601m 35% 10213Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4310Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11000m 36% 10370Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3936m 24% 6289Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 9490m 59% 14313Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10196m 64% 14351Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3760m 23% 6683Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 9474m 59% 14306Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7664m 48% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4693m 29% 7801Mi 13% 02:18:17 DEBUG --- stderr --- 02:18:17 DEBUG 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "0.021" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "1578.6666666666667" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "1557.8" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "ds-idrepo-1" 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG "pod": "ds-idrepo-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG }, 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "ds-idrepo-0" 02:19:00 DEBUG 02:19:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:00 DEBUG "pod": "ds-cts-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:00 DEBUG }, 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "0" 02:19:00 DEBUG 1712971140, 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG "282252" 02:19:00 DEBUG "29.566666666666666" 02:19:00 DEBUG ] 02:19:00 DEBUG "1557.8333333333333" 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG ] 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG ] 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "instance": "10.142.0.107:9100" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "instance": "10.142.0.77:9100" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "566758052864" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971140" 02:19:00 ERROR [http_cmd]: ERROR 02:19:00 ERROR --- status code --- 02:19:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:19:00 ERROR --- http response --- 02:19:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:19:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "instance": "10.142.0.107:9100" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "instance": "10.142.0.77:9100" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "46450630.4" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "ds-idrepo-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "ds-cts-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "0" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "ds-idrepo-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "ds-cts-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "0.0013957750000000001" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "1556.9666666666667" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "instance": "10.142.0.107:9100" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "instance": "10.142.0.77:9100" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "46447759.83333333" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "ds-idrepo-1" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "ds-idrepo-0" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "0" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "0" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:00 INFO [http_cmd]: http status code OK 02:19:00 DEBUG --- status code --- 02:19:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:00 DEBUG --- http response --- 02:19:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:00 DEBUG ----- output ----- 02:19:00 DEBUG { 02:19:00 DEBUG "status": "success", 02:19:00 DEBUG "data": { 02:19:00 DEBUG "resultType": "vector", 02:19:00 DEBUG "result": [ 02:19:00 DEBUG { 02:19:00 DEBUG "metric": { 02:19:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG [...] 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG 02:19:00 DEBUG "pod": "idm-b85f9d978-b559t" 02:19:00 DEBUG }, 02:19:00 DEBUG "value": [ 02:19:00 DEBUG 1712971140, 02:19:00 DEBUG "447.2470177162667" 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG ] 02:19:00 DEBUG } 02:19:00 DEBUG } 02:19:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971140" 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "pod": "ds-idrepo-1" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "pod": "ds-idrepo-0" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971140, 02:19:01 DEBUG "0" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "4.72400031834319" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "pod": "ds-idrepo-1" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "pod": "ds-cts-1" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "0" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "6236999680" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "instance": "10.142.0.107:9100" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "instance": "10.142.0.77:9100" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "0.11876666666665964" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "0" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "instance": "10.142.0.107:9100" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "instance": "10.142.0.77:9100" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "0.043000000000000434" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971141" 02:19:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971141" 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "0" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:01 INFO [http_cmd]: http status code OK 02:19:01 DEBUG --- status code --- 02:19:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:19:01 DEBUG --- http response --- 02:19:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:19:01 DEBUG ----- output ----- 02:19:01 DEBUG { 02:19:01 DEBUG "status": "success", 02:19:01 DEBUG "data": { 02:19:01 DEBUG "resultType": "vector", 02:19:01 DEBUG "result": [ 02:19:01 DEBUG { 02:19:01 DEBUG "metric": { 02:19:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG [...] 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG 02:19:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:19:01 DEBUG }, 02:19:01 DEBUG "value": [ 02:19:01 DEBUG 1712971141, 02:19:01 DEBUG "0.002948516365026061" 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG ] 02:19:01 DEBUG } 02:19:01 DEBUG } 02:19:12 INFO 02:19:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:19:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:19:12 INFO [loop_until]: OK (rc = 0) 02:19:12 DEBUG --- stdout --- 02:19:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 10511m 8532Mi am-6d96779b5d-qbg7l 8655m 8466Mi am-6d96779b5d-zn69c 10405m 8466Mi ds-cts-0 9703m 13450Mi ds-cts-1 9598m 13545Mi ds-cts-2 9716m 13452Mi ds-idrepo-0 7311m 5593Mi ds-idrepo-1 3755m 5522Mi ds-idrepo-2 3644m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 8m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 4875m 4064Mi 02:19:12 DEBUG --- stderr --- 02:19:12 DEBUG 02:19:17 INFO 02:19:17 INFO [loop_until]: kubectl --namespace=xlou top node 02:19:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:19:18 INFO [loop_until]: OK (rc = 0) 02:19:18 DEBUG --- stdout --- 02:19:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4743Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1855Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5048m 16% 10363Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 4233m 14% 3078Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 6002m 20% 10274Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4306Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 5881m 19% 10392Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 1372m 8% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 6636m 41% 14312Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 6005m 37% 14277Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 1863m 11% 6686Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 6846m 43% 14313Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 3664m 23% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 2862m 18% 7206Mi 12% 02:19:18 DEBUG --- stderr --- 02:19:18 DEBUG 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0.021" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "1567.5666666666666" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "instance": "10.142.0.107:9100" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "instance": "10.142.0.77:9100" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "567954789376" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971200" 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG "pod": "ds-idrepo-1" 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG }, 02:20:00 DEBUG }, 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG "value": [ 02:20:00 DEBUG "value": [ 02:20:00 DEBUG { 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "data": { 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG [...] 02:20:00 DEBUG [...] 02:20:00 DEBUG "result": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG { 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "metric": { 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "ds-idrepo-1" 02:20:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:00 DEBUG "pod": "ds-cts-1" 02:20:00 DEBUG }, 02:20:00 DEBUG }, 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG "value": [ 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG 02:20:00 DEBUG "239549.5" 02:20:00 DEBUG "29.666666666666668" 02:20:00 DEBUG 02:20:00 DEBUG ] 02:20:00 DEBUG ] 02:20:00 DEBUG [...] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 DEBUG 02:20:00 DEBUG ] 02:20:00 DEBUG ] 02:20:00 DEBUG 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 DEBUG 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 DEBUG "pod": "ds-idrepo-0" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 ERROR [http_cmd]: ERROR 02:20:00 ERROR --- status code --- 02:20:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:20:00 ERROR --- http response --- 02:20:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:20:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "instance": "10.142.0.107:9100" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "instance": "10.142.0.77:9100" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "10368097.666666666" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "ds-idrepo-1" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "ds-cts-1" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971200" 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "ds-idrepo-1" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "ds-cts-1" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0.001540991666666667" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG { 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "instance": "10.142.0.107:9100" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "instance": "10.142.0.77:9100" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "10364319.166666668" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "ds-idrepo-1" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "ds-idrepo-0" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971200" 02:20:00 INFO [http_cmd]: http status code OK 02:20:00 DEBUG --- status code --- 02:20:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:00 DEBUG --- http response --- 02:20:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:20:00 DEBUG ----- output ----- 02:20:00 DEBUG { 02:20:00 DEBUG "status": "success", 02:20:00 DEBUG "data": { 02:20:00 DEBUG "resultType": "vector", 02:20:00 DEBUG "result": [ 02:20:00 DEBUG { 02:20:00 DEBUG "metric": { 02:20:00 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG [...] 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG 02:20:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:00 DEBUG }, 02:20:00 DEBUG "value": [ 02:20:00 DEBUG 1712971200, 02:20:00 DEBUG "0" 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG ] 02:20:00 DEBUG } 02:20:00 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "544.9526026941626" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971201" 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971201" 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "pod": "ds-idrepo-1" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "pod": "ds-idrepo-0" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "pod": "ds-idrepo-1" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "pod": "ds-cts-1" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0.9388384679687556" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "6236864512" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "instance": "10.142.0.107:9100" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "instance": "10.142.0.77:9100" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0.01666666666666477" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0.002836786961583894" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971201" 02:20:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971201" 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "instance": "10.142.0.107:9100" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "instance": "10.142.0.77:9100" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0.004000000000000862" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:01 INFO [http_cmd]: http status code OK 02:20:01 DEBUG --- status code --- 02:20:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:20:01 DEBUG --- http response --- 02:20:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:20:01 DEBUG ----- output ----- 02:20:01 DEBUG { 02:20:01 DEBUG "status": "success", 02:20:01 DEBUG "data": { 02:20:01 DEBUG "resultType": "vector", 02:20:01 DEBUG "result": [ 02:20:01 DEBUG { 02:20:01 DEBUG "metric": { 02:20:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG [...] 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG 02:20:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:20:01 DEBUG }, 02:20:01 DEBUG "value": [ 02:20:01 DEBUG 1712971201, 02:20:01 DEBUG "0" 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG ] 02:20:01 DEBUG } 02:20:01 DEBUG } 02:20:12 INFO 02:20:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:20:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:20:12 INFO [loop_until]: OK (rc = 0) 02:20:12 DEBUG --- stdout --- 02:20:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 1775m 8527Mi am-6d96779b5d-qbg7l 1520m 8471Mi am-6d96779b5d-zn69c 1789m 8469Mi ds-cts-0 4307m 13448Mi ds-cts-1 4499m 13454Mi ds-cts-2 4238m 13450Mi ds-idrepo-0 864m 5593Mi ds-idrepo-1 357m 5522Mi ds-idrepo-2 348m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 1m 3460Mi 02:20:12 DEBUG --- stderr --- 02:20:12 DEBUG 02:20:18 INFO 02:20:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:20:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:20:18 INFO [loop_until]: OK (rc = 0) 02:20:18 DEBUG --- stdout --- 02:20:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4734Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1852Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1531m 5% 10350Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 3084Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1879m 6% 10247Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 133m 0% 4314Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1916m 6% 10363Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 398m 2% 6292Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4630m 29% 14325Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 4498m 28% 14293Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 411m 2% 6684Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 4731m 29% 14322Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 870m 5% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 7207Mi 12% 02:20:18 DEBUG --- stderr --- 02:20:18 DEBUG 02:20:59 WARNING File /tmp/lodemon_data-20240412_220143/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [13/Apr/2024 02:20:59] "GET /monitoring/average?start_time=20240413_002014&stop_time=20240413_011858 HTTP/1.1" 200 - 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0.021" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "instance": "10.142.0.107:9100" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "instance": "10.142.0.77:9100" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "568866638848" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "1573.6" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 ERROR [http_cmd]: ERROR 02:21:00 ERROR --- status code --- 02:21:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:21:00 ERROR --- http response --- 02:21:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:21:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "ds-idrepo-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "ds-idrepo-0" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "145431" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "ds-idrepo-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "ds-cts-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "29.733333333333334" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "ds-idrepo-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "ds-cts-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "instance": "10.142.0.107:9100" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "instance": "10.142.0.77:9100" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "10350679.066666666" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971260" 02:21:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971260" 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "ds-idrepo-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "ds-cts-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0.00023333333333333333" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "instance": "10.142.0.107:9100" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "instance": "10.142.0.77:9100" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "10346796.366666667" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:00 INFO [http_cmd]: http status code OK 02:21:00 DEBUG --- status code --- 02:21:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:00 DEBUG --- http response --- 02:21:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:00 DEBUG ----- output ----- 02:21:00 DEBUG { 02:21:00 DEBUG "status": "success", 02:21:00 DEBUG "data": { 02:21:00 DEBUG "resultType": "vector", 02:21:00 DEBUG "result": [ 02:21:00 DEBUG { 02:21:00 DEBUG "metric": { 02:21:00 DEBUG "pod": "ds-idrepo-1" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG [...] 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG 02:21:00 DEBUG "pod": "ds-idrepo-0" 02:21:00 DEBUG }, 02:21:00 DEBUG "value": [ 02:21:00 DEBUG 1712971260, 02:21:00 DEBUG "0" 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG ] 02:21:00 DEBUG } 02:21:00 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "idm-b85f9d978-dz57p" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "952.7836917314783" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0.024267013013554314" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "ds-idrepo-1" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "ds-idrepo-0" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "ds-idrepo-1" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "ds-cts-1" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "6236106752" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "instance": "10.142.0.107:9100" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "instance": "10.142.0.77:9100" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0.021966666666657867" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "idm-b85f9d978-dz57p" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971261" 02:21:01 INFO [http_cmd]: http status code OK 02:21:01 DEBUG --- status code --- 02:21:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:21:01 DEBUG --- http response --- 02:21:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:21:01 DEBUG ----- output ----- 02:21:01 DEBUG { 02:21:01 DEBUG "status": "success", 02:21:01 DEBUG "data": { 02:21:01 DEBUG "resultType": "vector", 02:21:01 DEBUG "result": [ 02:21:01 DEBUG { 02:21:01 DEBUG "metric": { 02:21:01 DEBUG "instance": "10.142.0.107:9100" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG [...] 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG 02:21:01 DEBUG "instance": "10.142.0.77:9100" 02:21:01 DEBUG }, 02:21:01 DEBUG "value": [ 02:21:01 DEBUG 1712971261, 02:21:01 DEBUG "0.0046666666666673965" 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG ] 02:21:01 DEBUG } 02:21:01 DEBUG } 02:21:12 INFO 02:21:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:21:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:21:12 INFO [loop_until]: OK (rc = 0) 02:21:12 DEBUG --- stdout --- 02:21:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 1792m 8485Mi am-6d96779b5d-qbg7l 1513m 8425Mi am-6d96779b5d-zn69c 1743m 8418Mi ds-cts-0 4278m 13366Mi ds-cts-1 4554m 13451Mi ds-cts-2 4447m 13448Mi ds-idrepo-0 853m 5593Mi ds-idrepo-1 346m 5521Mi ds-idrepo-2 388m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 1m 3460Mi 02:21:12 DEBUG --- stderr --- 02:21:12 DEBUG 02:21:18 INFO 02:21:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:21:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:21:18 INFO [loop_until]: OK (rc = 0) 02:21:18 DEBUG --- stdout --- 02:21:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4736Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1844Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1575m 5% 10346Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 289m 0% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1891m 6% 10245Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 4325Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1954m 6% 10343Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 441m 2% 6295Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4380m 27% 14197Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 4404m 27% 14302Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 418m 2% 6688Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 4292m 27% 13420Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 918m 5% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 517m 3% 7457Mi 12% 02:21:18 DEBUG --- stderr --- 02:21:18 DEBUG 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "0.021" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:00 DEBUG { 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG { 02:22:00 DEBUG "data": { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "result": [ 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG { 02:22:00 DEBUG "result": [ 02:22:00 DEBUG "metric": { 02:22:00 DEBUG { 02:22:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:00 DEBUG "metric": { 02:22:00 DEBUG }, 02:22:00 DEBUG "instance": "10.142.0.107:9100" 02:22:00 DEBUG "value": [ 02:22:00 DEBUG }, 02:22:00 DEBUG 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:00 DEBUG 02:22:00 DEBUG }, 02:22:00 DEBUG "instance": "10.142.0.77:9100" 02:22:00 DEBUG "value": [ 02:22:00 DEBUG }, 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG "710.8333333333334" 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG ] 02:22:00 DEBUG "569661350912" 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "1557.0666666666666" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971320" 02:22:00 ERROR [http_cmd]: ERROR 02:22:00 ERROR --- status code --- 02:22:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:22:00 ERROR --- http response --- 02:22:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:22:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "710.3666666666667" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "ds-idrepo-1" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "ds-idrepo-0" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "0" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "62084" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "ds-idrepo-1" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "ds-cts-1" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "29.9" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "ds-idrepo-1" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "ds-cts-1" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "0" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971320" 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971320" 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "711.1" 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 INFO [http_cmd]: http status code OK 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG --- status code --- 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG --- http response --- 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG ----- output ----- 02:22:00 DEBUG { 02:22:00 DEBUG { 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "status": "success", 02:22:00 DEBUG "data": { 02:22:00 DEBUG "data": { 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "resultType": "vector", 02:22:00 DEBUG "result": [ 02:22:00 DEBUG "result": [ 02:22:00 DEBUG { 02:22:00 DEBUG { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "metric": { 02:22:00 DEBUG "instance": "10.142.0.107:9100" 02:22:00 DEBUG "instance": "10.142.0.107:9100" 02:22:00 DEBUG }, 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG [...] 02:22:00 DEBUG [...] 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG 02:22:00 DEBUG "instance": "10.142.0.77:9100" 02:22:00 DEBUG "instance": "10.142.0.77:9100" 02:22:00 DEBUG }, 02:22:00 DEBUG }, 02:22:00 DEBUG "value": [ 02:22:00 DEBUG "value": [ 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG 1712971320, 02:22:00 DEBUG "29899601.866666667" 02:22:00 DEBUG "29905115.366666667" 02:22:00 DEBUG ] 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 DEBUG ] 02:22:00 DEBUG ] 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 DEBUG } 02:22:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971320" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "ds-idrepo-1" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "ds-cts-1" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971320, 02:22:01 DEBUG "0.0006497000000000001" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "ds-idrepo-1" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "ds-idrepo-0" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "2864747.755402299" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "4394.284137931035" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971321" 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "1.6679590024401671" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "ds-idrepo-1" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "ds-idrepo-0" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "ds-idrepo-1" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "ds-cts-1" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "6237949952" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "instance": "10.142.0.107:9100" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "instance": "10.142.0.77:9100" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0.0434333333333304" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0.003651652025861687" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971321" 02:22:01 INFO [http_cmd]: http status code OK 02:22:01 DEBUG --- status code --- 02:22:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:22:01 DEBUG --- http response --- 02:22:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:22:01 DEBUG ----- output ----- 02:22:01 DEBUG { 02:22:01 DEBUG "status": "success", 02:22:01 DEBUG "data": { 02:22:01 DEBUG "resultType": "vector", 02:22:01 DEBUG "result": [ 02:22:01 DEBUG { 02:22:01 DEBUG "metric": { 02:22:01 DEBUG "instance": "10.142.0.107:9100" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG [...] 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG 02:22:01 DEBUG "instance": "10.142.0.77:9100" 02:22:01 DEBUG }, 02:22:01 DEBUG "value": [ 02:22:01 DEBUG 1712971321, 02:22:01 DEBUG "0.02233333333333434" 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG ] 02:22:01 DEBUG } 02:22:01 DEBUG } 02:22:12 INFO 02:22:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:22:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:22:12 INFO [loop_until]: OK (rc = 0) 02:22:12 DEBUG --- stdout --- 02:22:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 6781m 8493Mi am-6d96779b5d-qbg7l 3642m 8428Mi am-6d96779b5d-zn69c 6962m 8429Mi ds-cts-0 6606m 13447Mi ds-cts-1 6197m 13097Mi ds-cts-2 6964m 12684Mi ds-idrepo-0 3229m 5593Mi ds-idrepo-1 1777m 5522Mi ds-idrepo-2 2024m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 10m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 3133m 3873Mi 02:22:12 DEBUG --- stderr --- 02:22:12 DEBUG 02:22:18 INFO 02:22:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:22:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:22:18 INFO [loop_until]: OK (rc = 0) 02:22:18 DEBUG --- stdout --- 02:22:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4742Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1853Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8886m 29% 10295Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13916m 46% 3079Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10201m 34% 10232Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4302Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10725m 35% 10314Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4111m 25% 6293Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 6959m 43% 13403Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7980m 50% 13603Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 3930m 24% 6684Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 6877m 43% 13696Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 7611m 47% 6373Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5204m 32% 7615Mi 12% 02:22:18 DEBUG --- stderr --- 02:22:18 DEBUG 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "0.021" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "1734.4333333333334" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971380" 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "instance": "10.142.0.107:9100" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "instance": "10.142.0.77:9100" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "571324419072" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "0" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971380" 02:23:00 ERROR [http_cmd]: ERROR 02:23:00 ERROR --- status code --- 02:23:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:23:00 ERROR --- http response --- 02:23:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:23:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "1734.5333333333333" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "ds-idrepo-1" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "ds-idrepo-0" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "0" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "99204.5" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971380" 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971380" 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "pod": "ds-idrepo-1" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "pod": "ds-cts-1" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "29.733333333333334" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:00 INFO [http_cmd]: http status code OK 02:23:00 DEBUG --- status code --- 02:23:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:00 DEBUG --- http response --- 02:23:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:00 DEBUG ----- output ----- 02:23:00 DEBUG { 02:23:00 DEBUG "status": "success", 02:23:00 DEBUG "data": { 02:23:00 DEBUG "resultType": "vector", 02:23:00 DEBUG "result": [ 02:23:00 DEBUG { 02:23:00 DEBUG "metric": { 02:23:00 DEBUG "instance": "10.142.0.107:9100" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG [...] 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG 02:23:00 DEBUG "instance": "10.142.0.77:9100" 02:23:00 DEBUG }, 02:23:00 DEBUG "value": [ 02:23:00 DEBUG 1712971380, 02:23:00 DEBUG "41474537.766666666" 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG ] 02:23:00 DEBUG } 02:23:00 DEBUG } 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971380, 02:23:01 DEBUG "1734.5666666666666" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "ds-idrepo-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "ds-cts-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "instance": "10.142.0.107:9100" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "instance": "10.142.0.77:9100" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "41469501.03333333" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "ds-idrepo-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "ds-cts-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0.000023163888888888888" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "ds-idrepo-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "ds-idrepo-0" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "276857858.1335324" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "4195352.551543988" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971381" 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "5.232809008996979" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "ds-idrepo-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "ds-idrepo-0" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "ds-idrepo-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "ds-cts-1" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "6237577216" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "instance": "10.142.0.107:9100" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "instance": "10.142.0.77:9100" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0.046199999999993226" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971381" 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971381" 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "instance": "10.142.0.107:9100" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG [...] 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "instance": "10.142.0.77:9100" 02:23:01 DEBUG }, 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG "0.025666666666666536" 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG --- http response --- 02:23:01 INFO [http_cmd]: http status code OK 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:23:01 DEBUG --- status code --- 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:23:01 DEBUG { 02:23:01 DEBUG --- http response --- 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "data": { 02:23:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG ----- output ----- 02:23:01 DEBUG "result": [ 02:23:01 DEBUG { 02:23:01 DEBUG { 02:23:01 DEBUG "status": "success", 02:23:01 DEBUG "metric": { 02:23:01 DEBUG "data": { 02:23:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:23:01 DEBUG "resultType": "vector", 02:23:01 DEBUG }, 02:23:01 DEBUG "result": [ 02:23:01 DEBUG "value": [ 02:23:01 DEBUG { 02:23:01 DEBUG 02:23:01 DEBUG "metric": { 02:23:01 DEBUG 02:23:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:23:01 DEBUG 02:23:01 DEBUG }, 02:23:01 DEBUG [...] 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG 02:23:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:01 DEBUG [...] 02:23:01 DEBUG }, 02:23:01 DEBUG 02:23:01 DEBUG "value": [ 02:23:01 DEBUG 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG 02:23:01 DEBUG "0" 02:23:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:23:01 DEBUG ] 02:23:01 DEBUG }, 02:23:01 DEBUG } 02:23:01 DEBUG "value": [ 02:23:01 DEBUG ] 02:23:01 DEBUG 1712971381, 02:23:01 DEBUG } 02:23:01 DEBUG "9.256847373822263" 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG ] 02:23:01 DEBUG } 02:23:01 DEBUG } 02:23:12 INFO 02:23:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:23:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:23:12 INFO [loop_until]: OK (rc = 0) 02:23:12 DEBUG --- stdout --- 02:23:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 9183m 8489Mi am-6d96779b5d-qbg7l 7854m 8434Mi am-6d96779b5d-zn69c 8992m 8443Mi ds-cts-0 5142m 13208Mi ds-cts-1 5210m 13290Mi ds-cts-2 4863m 12769Mi ds-idrepo-0 7554m 5593Mi ds-idrepo-1 3795m 5522Mi ds-idrepo-2 3704m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5309m 3879Mi 02:23:12 DEBUG --- stderr --- 02:23:12 DEBUG 02:23:18 INFO 02:23:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:23:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:23:18 INFO [loop_until]: OK (rc = 0) 02:23:18 DEBUG --- stdout --- 02:23:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4735Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1861Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7901m 26% 10289Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13862m 46% 3081Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9120m 30% 10228Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 4302Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9881m 33% 10311Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3925m 24% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4985m 31% 13264Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 5350m 33% 13507Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 3720m 23% 6684Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 5106m 32% 13578Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 7644m 48% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5360m 33% 7618Mi 12% 02:23:18 DEBUG --- stderr --- 02:23:18 DEBUG 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971440" 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971440" 02:24:00 INFO [http_cmd]: http status code OK 02:24:00 DEBUG --- status code --- 02:24:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:00 DEBUG --- http response --- 02:24:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:00 DEBUG ----- output ----- 02:24:00 DEBUG { 02:24:00 DEBUG "status": "success", 02:24:00 DEBUG "data": { 02:24:00 DEBUG "resultType": "vector", 02:24:00 DEBUG "result": [ 02:24:00 DEBUG { 02:24:00 DEBUG "metric": { 02:24:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG [...] 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 1712971440, 02:24:00 DEBUG "0.021" 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG } 02:24:00 INFO [http_cmd]: http status code OK 02:24:00 DEBUG --- status code --- 02:24:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:00 DEBUG --- http response --- 02:24:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971440" 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971440" 02:24:00 INFO [http_cmd]: http status code OK 02:24:00 DEBUG --- status code --- 02:24:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:00 DEBUG --- http response --- 02:24:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:00 DEBUG ----- output ----- 02:24:00 DEBUG { 02:24:00 DEBUG "status": "success", 02:24:00 DEBUG "data": { 02:24:00 DEBUG "resultType": "vector", 02:24:00 DEBUG "result": [ 02:24:00 DEBUG { 02:24:00 DEBUG "metric": { 02:24:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG [...] 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 1712971440, 02:24:00 DEBUG "0" 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG } 02:24:00 INFO [http_cmd]: http status code OK 02:24:00 DEBUG --- status code --- 02:24:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:00 DEBUG --- http response --- 02:24:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:00 DEBUG ----- output ----- 02:24:00 DEBUG { 02:24:00 DEBUG "status": "success", 02:24:00 DEBUG "data": { 02:24:00 DEBUG "resultType": "vector", 02:24:00 DEBUG "result": [ 02:24:00 DEBUG { 02:24:00 DEBUG "metric": { 02:24:00 DEBUG "instance": "10.142.0.107:9100" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG [...] 02:24:00 DEBUG 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971440" 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG "instance": "10.142.0.77:9100" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 1712971440, 02:24:00 DEBUG "573225741312" 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG } 02:24:00 INFO [http_cmd]: http status code OK 02:24:00 DEBUG --- status code --- 02:24:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:00 DEBUG --- http response --- 02:24:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:00 DEBUG ----- output ----- 02:24:00 DEBUG { 02:24:00 DEBUG "status": "success", 02:24:00 DEBUG "data": { 02:24:00 DEBUG "resultType": "vector", 02:24:00 DEBUG "result": [ 02:24:00 DEBUG { 02:24:00 DEBUG "metric": { 02:24:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG [...] 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG 02:24:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:00 DEBUG }, 02:24:00 DEBUG "value": [ 02:24:00 DEBUG 1712971440, 02:24:00 DEBUG "1738.4333333333334" 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG ] 02:24:00 DEBUG } 02:24:00 DEBUG } 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971440" 02:24:00 ERROR [http_cmd]: ERROR 02:24:00 ERROR --- status code --- 02:24:00 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:24:00 ERROR --- http response --- 02:24:00 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:24:00 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:24:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971440" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-idrepo-0" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971440, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:01 DEBUG }, 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:01 DEBUG }, 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "203656.5" 02:24:01 DEBUG "value": [ 02:24:01 DEBUG ] 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG } 02:24:01 DEBUG "1738.4333333333334" 02:24:01 DEBUG ] 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "1738.5" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "instance": "10.142.0.107:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "instance": "10.142.0.77:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "41282709.03333333" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-cts-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "29.633333333333333" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-cts-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "instance": "10.142.0.107:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "instance": "10.142.0.77:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "41277730.233333334" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-cts-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0.000023163888888888888" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-idrepo-0" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "1178.2488126566284" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "35794710.221941814" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-idrepo-0" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "5.236793305857564" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "ds-idrepo-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "ds-cts-1" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "instance": "10.142.0.107:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "instance": "10.142.0.77:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0.0519999999999977" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "6236831744" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0.0045784414730296406" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971441" 02:24:01 INFO [http_cmd]: http status code OK 02:24:01 DEBUG --- status code --- 02:24:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:24:01 DEBUG --- http response --- 02:24:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:24:01 DEBUG ----- output ----- 02:24:01 DEBUG { 02:24:01 DEBUG "status": "success", 02:24:01 DEBUG "data": { 02:24:01 DEBUG "resultType": "vector", 02:24:01 DEBUG "result": [ 02:24:01 DEBUG { 02:24:01 DEBUG "metric": { 02:24:01 DEBUG "instance": "10.142.0.107:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG [...] 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG 02:24:01 DEBUG "instance": "10.142.0.77:9100" 02:24:01 DEBUG }, 02:24:01 DEBUG "value": [ 02:24:01 DEBUG 1712971441, 02:24:01 DEBUG "0.02833333333333338" 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG ] 02:24:01 DEBUG } 02:24:01 DEBUG } 02:24:12 INFO 02:24:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:24:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:24:12 INFO [loop_until]: OK (rc = 0) 02:24:12 DEBUG --- stdout --- 02:24:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 9274m 8487Mi am-6d96779b5d-qbg7l 7813m 8429Mi am-6d96779b5d-zn69c 8967m 8429Mi ds-cts-0 5053m 13024Mi ds-cts-1 5402m 13115Mi ds-cts-2 4958m 12679Mi ds-idrepo-0 7536m 5593Mi ds-idrepo-1 3819m 5522Mi ds-idrepo-2 3836m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5083m 3883Mi 02:24:12 DEBUG --- stderr --- 02:24:12 DEBUG 02:24:18 INFO 02:24:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:24:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:24:18 INFO [loop_until]: OK (rc = 0) 02:24:18 DEBUG --- stdout --- 02:24:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4737Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 1852Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7919m 26% 10284Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14010m 46% 3097Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9233m 30% 10219Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 131m 0% 4307Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9801m 32% 10320Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 3898m 24% 6293Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 5318m 33% 13998Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 5807m 36% 14237Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 3798m 23% 6688Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 5367m 33% 14304Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7605m 47% 6372Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5382m 33% 7620Mi 12% 02:24:18 DEBUG --- stderr --- 02:24:18 DEBUG 02:25:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971500" 02:25:00 INFO [http_cmd]: http status code OK 02:25:00 DEBUG --- status code --- 02:25:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:00 DEBUG --- http response --- 02:25:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:25:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971500" 02:25:00 INFO [http_cmd]: http status code OK 02:25:00 DEBUG --- status code --- 02:25:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:00 DEBUG --- http response --- 02:25:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:00 DEBUG ----- output ----- 02:25:00 DEBUG { 02:25:00 DEBUG "status": "success", 02:25:00 DEBUG "data": { 02:25:00 DEBUG "resultType": "vector", 02:25:00 DEBUG "result": [ 02:25:00 DEBUG { 02:25:00 DEBUG "metric": { 02:25:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:00 DEBUG }, 02:25:00 DEBUG "value": [ 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG [...] 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:00 DEBUG }, 02:25:00 DEBUG "value": [ 02:25:00 DEBUG 1712971500, 02:25:00 DEBUG "0.021" 02:25:00 DEBUG ] 02:25:00 DEBUG } 02:25:00 DEBUG ] 02:25:00 DEBUG } 02:25:00 DEBUG } 02:25:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971500" 02:25:00 INFO [http_cmd]: http status code OK 02:25:00 DEBUG --- status code --- 02:25:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:00 DEBUG --- http response --- 02:25:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:00 DEBUG ----- output ----- 02:25:00 DEBUG { 02:25:00 DEBUG "status": "success", 02:25:00 DEBUG "data": { 02:25:00 DEBUG "resultType": "vector", 02:25:00 DEBUG "result": [ 02:25:00 DEBUG { 02:25:00 DEBUG "metric": { 02:25:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:00 DEBUG }, 02:25:00 DEBUG "value": [ 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG [...] 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:00 DEBUG }, 02:25:00 DEBUG "value": [ 02:25:00 DEBUG 1712971500, 02:25:00 DEBUG "707.6333333333333" 02:25:00 DEBUG ] 02:25:00 DEBUG } 02:25:00 DEBUG ] 02:25:00 DEBUG } 02:25:00 DEBUG } 02:25:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971500" 02:25:00 INFO [http_cmd]: http status code OK 02:25:00 DEBUG --- status code --- 02:25:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:00 DEBUG --- http response --- 02:25:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:00 DEBUG ----- output ----- 02:25:00 DEBUG { 02:25:00 DEBUG "status": "success", 02:25:00 DEBUG "data": { 02:25:00 DEBUG "resultType": "vector", 02:25:00 DEBUG "result": [ 02:25:00 DEBUG { 02:25:00 DEBUG "metric": { 02:25:00 DEBUG "instance": "10.142.0.107:9100" 02:25:00 DEBUG }, 02:25:00 DEBUG "value": [ 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG [...] 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG 02:25:00 DEBUG "instance": "10.142.0.77:9100" 02:25:00 DEBUG }, 02:25:00 DEBUG "value": [ 02:25:00 DEBUG 1712971500, 02:25:00 DEBUG "574988837888" 02:25:00 DEBUG ] 02:25:00 DEBUG } 02:25:00 DEBUG ] 02:25:00 DEBUG } 02:25:00 DEBUG } 02:25:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971500" 02:25:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971500" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971500, 02:25:01 DEBUG "1739.2333333333333" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 ERROR [http_cmd]: ERROR 02:25:01 ERROR --- status code --- 02:25:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:25:01 ERROR --- http response --- 02:25:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:25:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-idrepo-0" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "1738.1333333333332" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "297725" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "1740.0666666666666" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "instance": "10.142.0.107:9100" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "instance": "10.142.0.77:9100" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "46299809.56666666" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-cts-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG "0" 02:25:01 DEBUG { 02:25:01 DEBUG ] 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG } 02:25:01 DEBUG "data": { 02:25:01 DEBUG ] 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG } 02:25:01 DEBUG "result": [ 02:25:01 DEBUG } 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-cts-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "29.633333333333333" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-cts-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0.0004104527777777778" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "instance": "10.142.0.107:9100" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "instance": "10.142.0.77:9100" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "46294362.83333333" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-idrepo-0" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "30056.87512022273" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "5.228511395799513" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-idrepo-0" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "ds-idrepo-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "ds-cts-1" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971501" 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "instance": "10.142.0.107:9100" 02:25:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:25:01 DEBUG }, 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "instance": "10.142.0.77:9100" 02:25:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:25:01 DEBUG }, 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0.05053333333333162" 02:25:01 DEBUG "6238990336" 02:25:01 DEBUG ] 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "3.5413815785856264" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971501" 02:25:01 INFO [http_cmd]: http status code OK 02:25:01 DEBUG --- status code --- 02:25:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:25:01 DEBUG --- http response --- 02:25:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:25:01 DEBUG ----- output ----- 02:25:01 DEBUG { 02:25:01 DEBUG "status": "success", 02:25:01 DEBUG "data": { 02:25:01 DEBUG "resultType": "vector", 02:25:01 DEBUG "result": [ 02:25:01 DEBUG { 02:25:01 DEBUG "metric": { 02:25:01 DEBUG "instance": "10.142.0.107:9100" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG [...] 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG 02:25:01 DEBUG "instance": "10.142.0.77:9100" 02:25:01 DEBUG }, 02:25:01 DEBUG "value": [ 02:25:01 DEBUG 1712971501, 02:25:01 DEBUG "0.023000000000000402" 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG ] 02:25:01 DEBUG } 02:25:01 DEBUG } 02:25:12 INFO 02:25:12 INFO [loop_until]: kubectl --namespace=xlou top pods 02:25:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:25:12 INFO [loop_until]: OK (rc = 0) 02:25:12 DEBUG --- stdout --- 02:25:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11139m 8492Mi am-6d96779b5d-qbg7l 8881m 8435Mi am-6d96779b5d-zn69c 9818m 8437Mi ds-cts-0 7717m 13451Mi ds-cts-1 7823m 13455Mi ds-cts-2 7680m 13451Mi ds-idrepo-0 7618m 5593Mi ds-idrepo-1 4076m 5522Mi ds-idrepo-2 3948m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 12m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 7m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5207m 3886Mi 02:25:12 DEBUG --- stderr --- 02:25:12 DEBUG 02:25:18 INFO 02:25:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:25:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:25:18 INFO [loop_until]: OK (rc = 0) 02:25:18 DEBUG --- stdout --- 02:25:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4732Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1865Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9990m 33% 10302Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13577m 45% 3118Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11803m 39% 10229Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 4332Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12525m 41% 10323Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4216m 26% 6293Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10512m 66% 14300Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10613m 66% 14279Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4084m 25% 6685Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10664m 67% 14306Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8237m 51% 6370Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5385m 33% 7624Mi 13% 02:25:18 DEBUG --- stderr --- 02:25:18 DEBUG 02:26:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971560" 02:26:00 INFO [http_cmd]: http status code OK 02:26:00 DEBUG --- status code --- 02:26:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:00 DEBUG --- http response --- 02:26:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:26:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971560" 02:26:00 INFO [http_cmd]: http status code OK 02:26:00 DEBUG --- status code --- 02:26:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:00 DEBUG --- http response --- 02:26:00 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:00 DEBUG ----- output ----- 02:26:00 DEBUG { 02:26:00 DEBUG "status": "success", 02:26:00 DEBUG "data": { 02:26:00 DEBUG "resultType": "vector", 02:26:00 DEBUG "result": [ 02:26:00 DEBUG { 02:26:00 DEBUG "metric": { 02:26:00 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:00 DEBUG }, 02:26:00 DEBUG "value": [ 02:26:00 DEBUG 02:26:00 DEBUG 02:26:00 DEBUG 02:26:00 DEBUG [...] 02:26:00 DEBUG 02:26:00 DEBUG 02:26:00 DEBUG 02:26:00 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:00 DEBUG }, 02:26:00 DEBUG "value": [ 02:26:00 DEBUG 1712971560, 02:26:00 DEBUG "0.021" 02:26:00 DEBUG ] 02:26:00 DEBUG } 02:26:00 DEBUG ] 02:26:00 DEBUG } 02:26:00 DEBUG } 02:26:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971560" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971560, 02:26:01 DEBUG "1735.5" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "instance": "10.142.0.107:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "instance": "10.142.0.77:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "576954500096" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "1743.9333333333334" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 ERROR [http_cmd]: ERROR 02:26:01 ERROR --- status code --- 02:26:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:26:01 ERROR --- http response --- 02:26:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:26:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "1744.0333333333333" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-idrepo-0" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "1743.9666666666667" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "313875" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0.0005333333333333333" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "instance": "10.142.0.107:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "instance": "10.142.0.77:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "51320065.33333333" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "29.8" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "instance": "10.142.0.107:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "instance": "10.142.0.77:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "51314393.233333334" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-idrepo-0" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "22071300.91512966" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "455430.9990039499" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "5.257758868244355" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-idrepo-0" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "ds-idrepo-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "6239326208" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "instance": "10.142.0.107:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "instance": "10.142.0.77:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0.0580666666666725" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "41758490.87054784" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971561" 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "instance": "10.142.0.107:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "instance": "10.142.0.77:9100" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0.026333333333335908" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "ds-cts-1" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "7.2962513591372975" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971561" 02:26:01 INFO [http_cmd]: http status code OK 02:26:01 DEBUG --- status code --- 02:26:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:26:01 DEBUG --- http response --- 02:26:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:26:01 DEBUG ----- output ----- 02:26:01 DEBUG { 02:26:01 DEBUG "status": "success", 02:26:01 DEBUG "data": { 02:26:01 DEBUG "resultType": "vector", 02:26:01 DEBUG "result": [ 02:26:01 DEBUG { 02:26:01 DEBUG "metric": { 02:26:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG [...] 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG 02:26:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:26:01 DEBUG }, 02:26:01 DEBUG "value": [ 02:26:01 DEBUG 1712971561, 02:26:01 DEBUG "0" 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG ] 02:26:01 DEBUG } 02:26:01 DEBUG } 02:26:13 INFO 02:26:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:26:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:26:13 INFO [loop_until]: OK (rc = 0) 02:26:13 DEBUG --- stdout --- 02:26:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11723m 8516Mi am-6d96779b5d-qbg7l 9644m 8442Mi am-6d96779b5d-zn69c 11381m 8438Mi ds-cts-0 10539m 13479Mi ds-cts-1 10829m 13465Mi ds-cts-2 10589m 13452Mi ds-idrepo-0 8196m 5593Mi ds-idrepo-1 4396m 5522Mi ds-idrepo-2 4146m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5275m 3936Mi 02:26:13 DEBUG --- stderr --- 02:26:13 DEBUG 02:26:18 INFO 02:26:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:26:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:26:18 INFO [loop_until]: OK (rc = 0) 02:26:18 DEBUG --- stdout --- 02:26:18 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4729Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1867Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10086m 33% 10308Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14111m 47% 3125Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11778m 39% 10237Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4304Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12566m 42% 10339Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4354m 27% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10457m 65% 14335Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10861m 68% 14289Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4221m 26% 6687Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10508m 66% 14310Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8262m 51% 6373Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5398m 33% 7671Mi 13% 02:26:18 DEBUG --- stderr --- 02:26:18 DEBUG 02:27:00 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971620" 02:27:00 INFO [http_cmd]: http status code OK 02:27:00 DEBUG --- status code --- 02:27:00 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:00 DEBUG --- http response --- 02:27:00 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0.022" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "1736.1666666666667" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "instance": "10.142.0.107:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "instance": "10.142.0.77:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "579080766464" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971621" 02:27:01 ERROR [http_cmd]: ERROR 02:27:01 ERROR --- status code --- 02:27:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:27:01 ERROR --- http response --- 02:27:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:27:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "1751.6333333333332" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "1752" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "1751.3999999999999" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-idrepo-0" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-cts-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "314144" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-cts-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "29.599999999999998" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-cts-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0.0005333333333333334" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "instance": "10.142.0.107:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "instance": "10.142.0.77:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "51177684.06666666" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "instance": "10.142.0.107:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "instance": "10.142.0.77:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "51174572.266666666" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-idrepo-0" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "1155.4155392224493" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "13926.08508551934" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "5.242799595218077" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-cts-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "ds-idrepo-1" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "ds-idrepo-0" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "6237229056" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "instance": "10.142.0.107:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "instance": "10.142.0.77:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0.17873333333333694" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0.005382563373193963" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "instance": "10.142.0.107:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "instance": "10.142.0.77:9100" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0.057333333333333535" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971621" 02:27:01 INFO [http_cmd]: http status code OK 02:27:01 DEBUG --- status code --- 02:27:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:27:01 DEBUG --- http response --- 02:27:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:27:01 DEBUG ----- output ----- 02:27:01 DEBUG { 02:27:01 DEBUG "status": "success", 02:27:01 DEBUG "data": { 02:27:01 DEBUG "resultType": "vector", 02:27:01 DEBUG "result": [ 02:27:01 DEBUG { 02:27:01 DEBUG "metric": { 02:27:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG [...] 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG 02:27:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:27:01 DEBUG }, 02:27:01 DEBUG "value": [ 02:27:01 DEBUG 1712971621, 02:27:01 DEBUG "0" 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG ] 02:27:01 DEBUG } 02:27:01 DEBUG } 02:27:13 INFO 02:27:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:27:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:27:13 INFO [loop_until]: OK (rc = 0) 02:27:13 DEBUG --- stdout --- 02:27:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11852m 8515Mi am-6d96779b5d-qbg7l 9888m 8462Mi am-6d96779b5d-zn69c 11728m 8447Mi ds-cts-0 11047m 13502Mi ds-cts-1 10838m 13599Mi ds-cts-2 10212m 13461Mi ds-idrepo-0 8137m 5593Mi ds-idrepo-1 4382m 5522Mi ds-idrepo-2 4174m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5295m 3973Mi 02:27:13 DEBUG --- stderr --- 02:27:13 DEBUG 02:27:18 INFO 02:27:18 INFO [loop_until]: kubectl --namespace=xlou top node 02:27:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:27:19 INFO [loop_until]: OK (rc = 0) 02:27:19 DEBUG --- stdout --- 02:27:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4729Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9871m 33% 10333Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14048m 47% 3101Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11842m 39% 10270Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 4313Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12741m 42% 10389Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4409m 27% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10152m 63% 14326Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10724m 67% 14289Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4280m 26% 6684Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10703m 67% 14360Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8012m 50% 6374Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5397m 33% 7756Mi 13% 02:27:19 DEBUG --- stderr --- 02:27:19 DEBUG 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0.023" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "1783.1333333333332" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "instance": "10.142.0.107:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "instance": "10.142.0.77:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "581180662784" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971681" 02:28:01 ERROR [http_cmd]: ERROR 02:28:01 ERROR --- status code --- 02:28:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:28:01 ERROR --- http response --- 02:28:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:28:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "1752.5666666666666" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "ds-idrepo-0" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG [...] 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG 02:28:01 DEBUG { 02:28:01 DEBUG 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG 02:28:01 DEBUG "data": { 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG }, 02:28:01 DEBUG "result": [ 02:28:01 DEBUG "value": [ 02:28:01 DEBUG { 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "1746.2" 02:28:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:01 DEBUG ] 02:28:01 DEBUG }, 02:28:01 DEBUG } 02:28:01 DEBUG "value": [ 02:28:01 DEBUG ] 02:28:01 DEBUG 02:28:01 DEBUG } 02:28:01 DEBUG 02:28:01 DEBUG } 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "1752.2333333333333" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:01 DEBUG }, 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "ds-cts-1" 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "pod": "ds-cts-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0" 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "314632.5" 02:28:01 DEBUG ] 02:28:01 DEBUG "29.633333333333333" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG { 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:01 DEBUG "pod": "ds-cts-1" 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG }, 02:28:01 DEBUG { 02:28:01 DEBUG "value": [ 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "data": { 02:28:01 DEBUG "0.0007449166666666666" 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG ] 02:28:01 DEBUG "result": [ 02:28:01 DEBUG } 02:28:01 DEBUG { 02:28:01 DEBUG ] 02:28:01 DEBUG "metric": { 02:28:01 DEBUG } 02:28:01 DEBUG "instance": "10.142.0.107:9100" 02:28:01 DEBUG } 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "instance": "10.142.0.77:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "51163367.93333334" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "instance": "10.142.0.107:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "instance": "10.142.0.77:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "51158367.2" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "ds-idrepo-0" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "380258.76111544704" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "5070688.901396167" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971681" 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "5.259729268260347" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "ds-cts-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "ds-idrepo-1" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "ds-idrepo-0" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "instance": "10.142.0.107:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "instance": "10.142.0.77:9100" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0.16990000000000832" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "6236745728" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971681" 02:28:01 INFO [http_cmd]: http status code OK 02:28:01 DEBUG --- status code --- 02:28:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:01 DEBUG --- http response --- 02:28:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:28:01 DEBUG ----- output ----- 02:28:01 DEBUG { 02:28:01 DEBUG "status": "success", 02:28:01 DEBUG "data": { 02:28:01 DEBUG "resultType": "vector", 02:28:01 DEBUG "result": [ 02:28:01 DEBUG { 02:28:01 DEBUG "metric": { 02:28:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG [...] 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG 02:28:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:01 DEBUG }, 02:28:01 DEBUG "value": [ 02:28:01 DEBUG 1712971681, 02:28:01 DEBUG "0" 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG ] 02:28:01 DEBUG } 02:28:01 DEBUG } 02:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971682" 02:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971682" 02:28:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971682" 02:28:02 INFO [http_cmd]: http status code OK 02:28:02 DEBUG --- status code --- 02:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:02 DEBUG --- http response --- 02:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:28:02 DEBUG ----- output ----- 02:28:02 DEBUG { 02:28:02 DEBUG "status": "success", 02:28:02 DEBUG "data": { 02:28:02 DEBUG "resultType": "vector", 02:28:02 DEBUG "result": [ 02:28:02 DEBUG { 02:28:02 DEBUG "metric": { 02:28:02 DEBUG "instance": "10.142.0.107:9100" 02:28:02 DEBUG }, 02:28:02 DEBUG "value": [ 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG [...] 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG "instance": "10.142.0.77:9100" 02:28:02 DEBUG }, 02:28:02 DEBUG "value": [ 02:28:02 DEBUG 1712971682, 02:28:02 DEBUG "0.05033333333333421" 02:28:02 DEBUG ] 02:28:02 DEBUG } 02:28:02 DEBUG ] 02:28:02 DEBUG } 02:28:02 DEBUG } 02:28:02 INFO [http_cmd]: http status code OK 02:28:02 DEBUG --- status code --- 02:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:02 DEBUG --- http response --- 02:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:28:02 DEBUG ----- output ----- 02:28:02 DEBUG { 02:28:02 DEBUG "status": "success", 02:28:02 DEBUG "data": { 02:28:02 DEBUG "resultType": "vector", 02:28:02 DEBUG "result": [ 02:28:02 DEBUG { 02:28:02 DEBUG "metric": { 02:28:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:28:02 DEBUG }, 02:28:02 DEBUG "value": [ 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG [...] 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:02 DEBUG }, 02:28:02 DEBUG "value": [ 02:28:02 DEBUG 1712971682, 02:28:02 DEBUG "0" 02:28:02 DEBUG ] 02:28:02 DEBUG } 02:28:02 DEBUG ] 02:28:02 DEBUG } 02:28:02 DEBUG } 02:28:02 INFO [http_cmd]: http status code OK 02:28:02 DEBUG --- status code --- 02:28:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:28:02 DEBUG --- http response --- 02:28:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:28:02 DEBUG ----- output ----- 02:28:02 DEBUG { 02:28:02 DEBUG "status": "success", 02:28:02 DEBUG "data": { 02:28:02 DEBUG "resultType": "vector", 02:28:02 DEBUG "result": [ 02:28:02 DEBUG { 02:28:02 DEBUG "metric": { 02:28:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:28:02 DEBUG }, 02:28:02 DEBUG "value": [ 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG [...] 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG 02:28:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:28:02 DEBUG }, 02:28:02 DEBUG "value": [ 02:28:02 DEBUG 1712971682, 02:28:02 DEBUG "8.291123642775528" 02:28:02 DEBUG ] 02:28:02 DEBUG } 02:28:02 DEBUG ] 02:28:02 DEBUG } 02:28:02 DEBUG } 02:28:13 INFO 02:28:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:28:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:28:13 INFO [loop_until]: OK (rc = 0) 02:28:13 DEBUG --- stdout --- 02:28:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11972m 8546Mi am-6d96779b5d-qbg7l 9607m 8463Mi am-6d96779b5d-zn69c 11401m 8469Mi ds-cts-0 10821m 13484Mi ds-cts-1 10498m 13469Mi ds-cts-2 10802m 13484Mi ds-idrepo-0 8189m 5593Mi ds-idrepo-1 4189m 5522Mi ds-idrepo-2 4184m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5184m 4018Mi 02:28:13 DEBUG --- stderr --- 02:28:13 DEBUG 02:28:19 INFO 02:28:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:28:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:28:19 INFO [loop_until]: OK (rc = 0) 02:28:19 DEBUG --- stdout --- 02:28:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 4731Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1854Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10141m 33% 10333Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14077m 47% 3095Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12016m 40% 10279Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 148m 0% 4313Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12268m 41% 10381Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4417m 27% 6295Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10759m 67% 14367Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10995m 69% 14316Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4262m 26% 6686Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10802m 67% 14359Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8272m 52% 6375Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5381m 33% 7758Mi 13% 02:28:19 DEBUG --- stderr --- 02:28:19 DEBUG 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "instance": "10.142.0.107:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "instance": "10.142.0.77:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "583378603008" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0.023" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "1739.8333333333333" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 ERROR [http_cmd]: ERROR 02:29:01 ERROR --- status code --- 02:29:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:29:01 ERROR --- http response --- 02:29:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:29:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-idrepo-0" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "1748.6666666666667" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "1748.9666666666667" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "1751.7333333333333" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG { 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:29:01 DEBUG "data": { 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG }, 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "value": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG [...] 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG }, 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-cts-1" 02:29:01 DEBUG "315035.5" 02:29:01 DEBUG "pod": "ds-cts-1" 02:29:01 DEBUG }, 02:29:01 DEBUG ] 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG } 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG ] 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "29.466666666666665" 02:29:01 DEBUG } 02:29:01 DEBUG "0" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "instance": "10.142.0.107:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "instance": "10.142.0.77:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "51730273.93333334" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-cts-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0.00031497222222222224" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "instance": "10.142.0.107:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "instance": "10.142.0.77:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "51724877.96666667" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-idrepo-0" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:29:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:29:01 DEBUG }, 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "3076.4398620736747" 02:29:01 DEBUG "197745164.0435629" 02:29:01 DEBUG ] 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971741" 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "5.255345067484345" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-cts-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-idrepo-0" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "instance": "10.142.0.107:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "instance": "10.142.0.77:9100" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0.0807666666666672" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "6237036544" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971741" 02:29:01 INFO [http_cmd]: http status code OK 02:29:01 DEBUG --- status code --- 02:29:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:01 DEBUG --- http response --- 02:29:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:29:01 DEBUG ----- output ----- 02:29:01 DEBUG { 02:29:01 DEBUG "status": "success", 02:29:01 DEBUG "data": { 02:29:01 DEBUG "resultType": "vector", 02:29:01 DEBUG "result": [ 02:29:01 DEBUG { 02:29:01 DEBUG "metric": { 02:29:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG [...] 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG 02:29:01 DEBUG "pod": "ds-idrepo-1" 02:29:01 DEBUG }, 02:29:01 DEBUG "value": [ 02:29:01 DEBUG 1712971741, 02:29:01 DEBUG "0" 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG ] 02:29:01 DEBUG } 02:29:01 DEBUG } 02:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971742" 02:29:02 INFO [http_cmd]: http status code OK 02:29:02 DEBUG --- status code --- 02:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:02 DEBUG --- http response --- 02:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:29:02 DEBUG ----- output ----- 02:29:02 DEBUG { 02:29:02 DEBUG "status": "success", 02:29:02 DEBUG "data": { 02:29:02 DEBUG "resultType": "vector", 02:29:02 DEBUG "result": [ 02:29:02 DEBUG { 02:29:02 DEBUG "metric": { 02:29:02 DEBUG "instance": "10.142.0.107:9100" 02:29:02 DEBUG }, 02:29:02 DEBUG "value": [ 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG [...] 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG "instance": "10.142.0.77:9100" 02:29:02 DEBUG }, 02:29:02 DEBUG "value": [ 02:29:02 DEBUG 1712971742, 02:29:02 DEBUG "0.03266666666666656" 02:29:02 DEBUG ] 02:29:02 DEBUG } 02:29:02 DEBUG ] 02:29:02 DEBUG } 02:29:02 DEBUG } 02:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971742" 02:29:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971742" 02:29:02 INFO [http_cmd]: http status code OK 02:29:02 DEBUG --- status code --- 02:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:02 DEBUG --- http response --- 02:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:29:02 DEBUG ----- output ----- 02:29:02 DEBUG { 02:29:02 DEBUG "status": "success", 02:29:02 DEBUG "data": { 02:29:02 DEBUG "resultType": "vector", 02:29:02 DEBUG "result": [ 02:29:02 DEBUG { 02:29:02 DEBUG "metric": { 02:29:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:29:02 DEBUG }, 02:29:02 DEBUG "value": [ 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG [...] 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:29:02 DEBUG }, 02:29:02 DEBUG "value": [ 02:29:02 DEBUG 1712971742, 02:29:02 DEBUG "0" 02:29:02 DEBUG ] 02:29:02 DEBUG } 02:29:02 DEBUG ] 02:29:02 DEBUG } 02:29:02 DEBUG } 02:29:02 INFO [http_cmd]: http status code OK 02:29:02 DEBUG --- status code --- 02:29:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:29:02 DEBUG --- http response --- 02:29:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:29:02 DEBUG ----- output ----- 02:29:02 DEBUG { 02:29:02 DEBUG "status": "success", 02:29:02 DEBUG "data": { 02:29:02 DEBUG "resultType": "vector", 02:29:02 DEBUG "result": [ 02:29:02 DEBUG { 02:29:02 DEBUG "metric": { 02:29:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:29:02 DEBUG }, 02:29:02 DEBUG "value": [ 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG [...] 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG 02:29:02 DEBUG "pod": "ds-idrepo-1" 02:29:02 DEBUG }, 02:29:02 DEBUG "value": [ 02:29:02 DEBUG 1712971742, 02:29:02 DEBUG "4.343379987163297" 02:29:02 DEBUG ] 02:29:02 DEBUG } 02:29:02 DEBUG ] 02:29:02 DEBUG } 02:29:02 DEBUG } 02:29:13 INFO 02:29:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:29:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:29:13 INFO [loop_until]: OK (rc = 0) 02:29:13 DEBUG --- stdout --- 02:29:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12221m 8541Mi am-6d96779b5d-qbg7l 9560m 8465Mi am-6d96779b5d-zn69c 11379m 8478Mi ds-cts-0 10578m 13484Mi ds-cts-1 11091m 13483Mi ds-cts-2 10714m 13489Mi ds-idrepo-0 8214m 5594Mi ds-idrepo-1 4306m 5522Mi ds-idrepo-2 4156m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5352m 4019Mi 02:29:13 DEBUG --- stderr --- 02:29:13 DEBUG 02:29:19 INFO 02:29:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:29:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:29:19 INFO [loop_until]: OK (rc = 0) 02:29:19 DEBUG --- stdout --- 02:29:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4733Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1849Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10374m 34% 10341Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13671m 45% 3088Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12012m 40% 10277Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4305Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12767m 42% 10383Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4286m 26% 6293Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10827m 68% 14490Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11261m 70% 14329Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4122m 25% 6687Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10712m 67% 14354Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8282m 52% 6374Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5420m 34% 7757Mi 13% 02:29:19 DEBUG --- stderr --- 02:29:19 DEBUG 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "instance": "10.142.0.107:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "instance": "10.142.0.77:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "585793491968" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971801" 02:30:01 ERROR [http_cmd]: ERROR 02:30:01 ERROR --- status code --- 02:30:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:30:01 ERROR --- http response --- 02:30:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:30:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0.023" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-idrepo-0" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "1751.1333333333332" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "1731.4666666666667" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "1730.8333333333333" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "1731.9333333333334" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "instance": "10.142.0.107:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "instance": "10.142.0.77:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "51381130.86666667" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "315076.5" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-cts-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-cts-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "29.466666666666665" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-cts-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0.00003333333333333336" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "instance": "10.142.0.107:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "instance": "10.142.0.77:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "51375989.5" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-idrepo-0" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "549.098465044574" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-cts-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "ds-idrepo-1" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "ds-idrepo-0" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "5.277880594306634" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "instance": "10.142.0.107:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "instance": "10.142.0.77:9100" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0.188033333333343" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "6237257728" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971801" 02:30:01 INFO [http_cmd]: http status code OK 02:30:01 DEBUG --- status code --- 02:30:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:01 DEBUG --- http response --- 02:30:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:30:01 DEBUG ----- output ----- 02:30:01 DEBUG { 02:30:01 DEBUG "status": "success", 02:30:01 DEBUG "data": { 02:30:01 DEBUG "resultType": "vector", 02:30:01 DEBUG "result": [ 02:30:01 DEBUG { 02:30:01 DEBUG "metric": { 02:30:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG [...] 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG 02:30:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:30:01 DEBUG }, 02:30:01 DEBUG "value": [ 02:30:01 DEBUG 1712971801, 02:30:01 DEBUG "0" 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG ] 02:30:01 DEBUG } 02:30:01 DEBUG } 02:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971802" 02:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971802" 02:30:02 INFO [http_cmd]: http status code OK 02:30:02 DEBUG --- status code --- 02:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:02 DEBUG --- http response --- 02:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:02 DEBUG ----- output ----- 02:30:02 DEBUG { 02:30:02 DEBUG "status": "success", 02:30:02 DEBUG "data": { 02:30:02 DEBUG "resultType": "vector", 02:30:02 DEBUG "result": [ 02:30:02 DEBUG { 02:30:02 DEBUG "metric": { 02:30:02 DEBUG "instance": "10.142.0.107:9100" 02:30:02 DEBUG }, 02:30:02 DEBUG "value": [ 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG [...] 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG "instance": "10.142.0.77:9100" 02:30:02 DEBUG }, 02:30:02 DEBUG "value": [ 02:30:02 DEBUG 1712971802, 02:30:02 DEBUG "0.06566666666666711" 02:30:02 DEBUG ] 02:30:02 DEBUG } 02:30:02 DEBUG ] 02:30:02 DEBUG } 02:30:02 DEBUG } 02:30:02 INFO [http_cmd]: http status code OK 02:30:02 DEBUG --- status code --- 02:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:02 DEBUG --- http response --- 02:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:30:02 DEBUG ----- output ----- 02:30:02 DEBUG { 02:30:02 DEBUG "status": "success", 02:30:02 DEBUG "data": { 02:30:02 DEBUG "resultType": "vector", 02:30:02 DEBUG "result": [ 02:30:02 DEBUG { 02:30:02 DEBUG "metric": { 02:30:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:30:02 DEBUG }, 02:30:02 DEBUG "value": [ 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG [...] 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:30:02 DEBUG }, 02:30:02 DEBUG "value": [ 02:30:02 DEBUG 1712971802, 02:30:02 DEBUG "0" 02:30:02 DEBUG ] 02:30:02 DEBUG } 02:30:02 DEBUG ] 02:30:02 DEBUG } 02:30:02 DEBUG } 02:30:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971802" 02:30:02 INFO [http_cmd]: http status code OK 02:30:02 DEBUG --- status code --- 02:30:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:30:02 DEBUG --- http response --- 02:30:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:30:02 DEBUG ----- output ----- 02:30:02 DEBUG { 02:30:02 DEBUG "status": "success", 02:30:02 DEBUG "data": { 02:30:02 DEBUG "resultType": "vector", 02:30:02 DEBUG "result": [ 02:30:02 DEBUG { 02:30:02 DEBUG "metric": { 02:30:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:30:02 DEBUG }, 02:30:02 DEBUG "value": [ 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG [...] 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG 02:30:02 DEBUG "pod": "idm-b85f9d978-dz57p" 02:30:02 DEBUG }, 02:30:02 DEBUG "value": [ 02:30:02 DEBUG 1712971802, 02:30:02 DEBUG "0.005781770895837318" 02:30:02 DEBUG ] 02:30:02 DEBUG } 02:30:02 DEBUG ] 02:30:02 DEBUG } 02:30:02 DEBUG } 02:30:13 INFO 02:30:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:30:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:30:13 INFO [loop_until]: OK (rc = 0) 02:30:13 DEBUG --- stdout --- 02:30:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12137m 8542Mi am-6d96779b5d-qbg7l 10065m 8469Mi am-6d96779b5d-zn69c 11670m 8470Mi ds-cts-0 10923m 13483Mi ds-cts-1 11326m 13402Mi ds-cts-2 10684m 13489Mi ds-idrepo-0 8143m 5593Mi ds-idrepo-1 4185m 5522Mi ds-idrepo-2 4158m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1094Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5195m 4019Mi 02:30:13 DEBUG --- stderr --- 02:30:13 DEBUG 02:30:19 INFO 02:30:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:30:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:30:19 INFO [loop_until]: OK (rc = 0) 02:30:19 DEBUG --- stdout --- 02:30:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4737Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1849Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10266m 34% 10344Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14010m 46% 3086Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12029m 40% 10269Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 4317Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12717m 42% 10387Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4357m 27% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10831m 68% 14377Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11199m 70% 14340Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4193m 26% 6686Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10902m 68% 13788Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 8253m 51% 6376Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5348m 33% 7754Mi 13% 02:30:19 DEBUG --- stderr --- 02:30:19 DEBUG 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "instance": "10.142.0.107:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "instance": "10.142.0.77:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "588586089472" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-idrepo-0" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "1763.5666666666666" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971861" 02:31:01 ERROR [http_cmd]: ERROR 02:31:01 ERROR --- status code --- 02:31:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:31:01 ERROR --- http response --- 02:31:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:31:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0.023" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "1751.8999999999999" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "1746.9666666666667" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "1751.7" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "314377.5" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "instance": "10.142.0.107:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "instance": "10.142.0.77:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "51194968.93333334" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-cts-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-cts-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "28.433333333333334" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-cts-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0.0010525444444444444" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG } 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "instance": "10.142.0.107:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "instance": "10.142.0.77:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "51189917.666666664" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-idrepo-0" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:31:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:31:01 DEBUG }, 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:31:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG }, 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG "926.6134274049941" 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG ] 02:31:01 DEBUG "65092852.862301916" 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-cts-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971861" 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "pod": "ds-idrepo-1" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "pod": "ds-idrepo-0" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "5.222480766952799" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "instance": "10.142.0.107:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "instance": "10.142.0.77:9100" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "0.22356666666667735" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971861" 02:31:01 INFO [http_cmd]: http status code OK 02:31:01 DEBUG --- status code --- 02:31:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:01 DEBUG --- http response --- 02:31:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:01 DEBUG ----- output ----- 02:31:01 DEBUG { 02:31:01 DEBUG "status": "success", 02:31:01 DEBUG "data": { 02:31:01 DEBUG "resultType": "vector", 02:31:01 DEBUG "result": [ 02:31:01 DEBUG { 02:31:01 DEBUG "metric": { 02:31:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG [...] 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG 02:31:01 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:31:01 DEBUG }, 02:31:01 DEBUG "value": [ 02:31:01 DEBUG 1712971861, 02:31:01 DEBUG "6236614656" 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG ] 02:31:01 DEBUG } 02:31:01 DEBUG } 02:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971862" 02:31:02 INFO [http_cmd]: http status code OK 02:31:02 DEBUG --- status code --- 02:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:02 DEBUG --- http response --- 02:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:02 DEBUG ----- output ----- 02:31:02 DEBUG { 02:31:02 DEBUG "status": "success", 02:31:02 DEBUG "data": { 02:31:02 DEBUG "resultType": "vector", 02:31:02 DEBUG "result": [ 02:31:02 DEBUG { 02:31:02 DEBUG "metric": { 02:31:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG [...] 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 1712971862, 02:31:02 DEBUG "0" 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG } 02:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971862" 02:31:02 INFO [http_cmd]: http status code OK 02:31:02 DEBUG --- status code --- 02:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:02 DEBUG --- http response --- 02:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:31:02 DEBUG ----- output ----- 02:31:02 DEBUG { 02:31:02 DEBUG "status": "success", 02:31:02 DEBUG "data": { 02:31:02 DEBUG "resultType": "vector", 02:31:02 DEBUG "result": [ 02:31:02 DEBUG { 02:31:02 DEBUG "metric": { 02:31:02 DEBUG "instance": "10.142.0.107:9100" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG [...] 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG "instance": "10.142.0.77:9100" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 1712971862, 02:31:02 DEBUG "0.07300000000000018" 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG } 02:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971862" 02:31:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971862" 02:31:02 INFO [http_cmd]: http status code OK 02:31:02 DEBUG --- status code --- 02:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:02 DEBUG --- http response --- 02:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:31:02 DEBUG ----- output ----- 02:31:02 DEBUG { 02:31:02 DEBUG "status": "success", 02:31:02 DEBUG "data": { 02:31:02 DEBUG "resultType": "vector", 02:31:02 DEBUG "result": [ 02:31:02 DEBUG { 02:31:02 DEBUG "metric": { 02:31:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG [...] 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 1712971862, 02:31:02 DEBUG "0" 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG } 02:31:02 INFO [http_cmd]: http status code OK 02:31:02 DEBUG --- status code --- 02:31:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:31:02 DEBUG --- http response --- 02:31:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:31:02 DEBUG ----- output ----- 02:31:02 DEBUG { 02:31:02 DEBUG "status": "success", 02:31:02 DEBUG "data": { 02:31:02 DEBUG "resultType": "vector", 02:31:02 DEBUG "result": [ 02:31:02 DEBUG { 02:31:02 DEBUG "metric": { 02:31:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG [...] 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG 02:31:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:31:02 DEBUG }, 02:31:02 DEBUG "value": [ 02:31:02 DEBUG 1712971862, 02:31:02 DEBUG "0" 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG ] 02:31:02 DEBUG } 02:31:02 DEBUG } 02:31:13 INFO 02:31:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:31:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:31:13 INFO [loop_until]: OK (rc = 0) 02:31:13 DEBUG --- stdout --- 02:31:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11936m 8541Mi am-6d96779b5d-qbg7l 9835m 8468Mi am-6d96779b5d-zn69c 11377m 8471Mi ds-cts-0 10840m 13118Mi ds-cts-1 10961m 13512Mi ds-cts-2 10870m 13500Mi ds-idrepo-0 8159m 5593Mi ds-idrepo-1 4225m 5522Mi ds-idrepo-2 4156m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5368m 4020Mi 02:31:13 DEBUG --- stderr --- 02:31:13 DEBUG 02:31:19 INFO 02:31:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:31:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:31:19 INFO [loop_until]: OK (rc = 0) 02:31:19 DEBUG --- stdout --- 02:31:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4735Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1840Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10337m 34% 10341Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13995m 46% 3084Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11929m 39% 10271Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4304Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12719m 42% 10375Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4336m 27% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10834m 68% 14206Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11365m 71% 14444Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4212m 26% 6690Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10917m 68% 14366Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8231m 51% 6375Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5416m 34% 7757Mi 13% 02:31:19 DEBUG --- stderr --- 02:31:19 DEBUG 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971921" 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:01 DEBUG "result": [ 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:01 DEBUG "data": { 02:32:01 DEBUG }, 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "value": [ 02:32:01 DEBUG "result": [ 02:32:01 DEBUG 02:32:01 DEBUG { 02:32:01 DEBUG 02:32:01 DEBUG "metric": { 02:32:01 DEBUG 02:32:01 DEBUG "instance": "10.142.0.107:9100" 02:32:01 DEBUG [...] 02:32:01 DEBUG }, 02:32:01 DEBUG 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:01 DEBUG 02:32:01 DEBUG }, 02:32:01 DEBUG [...] 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG 02:32:01 DEBUG "1755.2666666666667" 02:32:01 DEBUG 02:32:01 DEBUG ] 02:32:01 DEBUG "instance": "10.142.0.77:9100" 02:32:01 DEBUG } 02:32:01 DEBUG }, 02:32:01 DEBUG ] 02:32:01 DEBUG "value": [ 02:32:01 DEBUG } 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG } 02:32:01 DEBUG "591273927680" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-idrepo-0" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971921" 02:32:01 ERROR [http_cmd]: ERROR 02:32:01 ERROR --- status code --- 02:32:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:32:01 ERROR --- http response --- 02:32:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:32:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0.023" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "1742.6" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "1743.7666666666667" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "1743.6333333333332" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "313901" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-cts-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "instance": "10.142.0.107:9100" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "instance": "10.142.0.77:9100" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "51251129.63333333" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-cts-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "29.133333333333333" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-cts-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0.00041129999999999997" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "instance": "10.142.0.107:9100" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "instance": "10.142.0.77:9100" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "51247948.1" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-idrepo-0" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "idm-b85f9d978-dz57p" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "1078.3586611481164" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-cts-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "pod": "ds-idrepo-1" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "pod": "ds-idrepo-0" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "5.235048575290894" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971921" 02:32:01 INFO [http_cmd]: http status code OK 02:32:01 DEBUG --- status code --- 02:32:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:01 DEBUG --- http response --- 02:32:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:01 DEBUG ----- output ----- 02:32:01 DEBUG { 02:32:01 DEBUG "status": "success", 02:32:01 DEBUG "data": { 02:32:01 DEBUG "resultType": "vector", 02:32:01 DEBUG "result": [ 02:32:01 DEBUG { 02:32:01 DEBUG "metric": { 02:32:01 DEBUG "instance": "10.142.0.107:9100" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG [...] 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG 02:32:01 DEBUG "instance": "10.142.0.77:9100" 02:32:01 DEBUG }, 02:32:01 DEBUG "value": [ 02:32:01 DEBUG 1712971921, 02:32:01 DEBUG "0.15986666666667637" 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG ] 02:32:01 DEBUG } 02:32:01 DEBUG } 02:32:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971921" 02:32:02 INFO [http_cmd]: http status code OK 02:32:02 DEBUG --- status code --- 02:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:02 DEBUG --- http response --- 02:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:02 DEBUG ----- output ----- 02:32:02 DEBUG { 02:32:02 DEBUG "status": "success", 02:32:02 DEBUG "data": { 02:32:02 DEBUG "resultType": "vector", 02:32:02 DEBUG "result": [ 02:32:02 DEBUG { 02:32:02 DEBUG "metric": { 02:32:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG [...] 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 1712971921, 02:32:02 DEBUG "6237167616" 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG } 02:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971922" 02:32:02 INFO [http_cmd]: http status code OK 02:32:02 DEBUG --- status code --- 02:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:02 DEBUG --- http response --- 02:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:02 DEBUG ----- output ----- 02:32:02 DEBUG { 02:32:02 DEBUG "status": "success", 02:32:02 DEBUG "data": { 02:32:02 DEBUG "resultType": "vector", 02:32:02 DEBUG "result": [ 02:32:02 DEBUG { 02:32:02 DEBUG "metric": { 02:32:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG [...] 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG "pod": "idm-b85f9d978-dz57p" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 1712971922, 02:32:02 DEBUG "0" 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG } 02:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971922" 02:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971922" 02:32:02 INFO [http_cmd]: http status code OK 02:32:02 DEBUG --- status code --- 02:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:02 DEBUG --- http response --- 02:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:32:02 DEBUG ----- output ----- 02:32:02 DEBUG { 02:32:02 DEBUG "status": "success", 02:32:02 DEBUG "data": { 02:32:02 DEBUG "resultType": "vector", 02:32:02 DEBUG "result": [ 02:32:02 DEBUG { 02:32:02 DEBUG "metric": { 02:32:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG [...] 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 1712971922, 02:32:02 DEBUG "0" 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG } 02:32:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971922" 02:32:02 INFO [http_cmd]: http status code OK 02:32:02 DEBUG --- status code --- 02:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:02 DEBUG --- http response --- 02:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:32:02 DEBUG ----- output ----- 02:32:02 DEBUG { 02:32:02 DEBUG "status": "success", 02:32:02 DEBUG "data": { 02:32:02 DEBUG "resultType": "vector", 02:32:02 DEBUG "result": [ 02:32:02 DEBUG { 02:32:02 DEBUG "metric": { 02:32:02 DEBUG "instance": "10.142.0.107:9100" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG [...] 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG "instance": "10.142.0.77:9100" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 1712971922, 02:32:02 DEBUG "0.05533333333333323" 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG } 02:32:02 INFO [http_cmd]: http status code OK 02:32:02 DEBUG --- status code --- 02:32:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:32:02 DEBUG --- http response --- 02:32:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:32:02 DEBUG ----- output ----- 02:32:02 DEBUG { 02:32:02 DEBUG "status": "success", 02:32:02 DEBUG "data": { 02:32:02 DEBUG "resultType": "vector", 02:32:02 DEBUG "result": [ 02:32:02 DEBUG { 02:32:02 DEBUG "metric": { 02:32:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG [...] 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG 02:32:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:32:02 DEBUG }, 02:32:02 DEBUG "value": [ 02:32:02 DEBUG 1712971922, 02:32:02 DEBUG "0" 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG ] 02:32:02 DEBUG } 02:32:02 DEBUG } 02:32:13 INFO 02:32:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:32:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:32:13 INFO [loop_until]: OK (rc = 0) 02:32:13 DEBUG --- stdout --- 02:32:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12244m 8541Mi am-6d96779b5d-qbg7l 9735m 8466Mi am-6d96779b5d-zn69c 11894m 8487Mi ds-cts-0 10571m 13489Mi ds-cts-1 10722m 13509Mi ds-cts-2 10594m 13499Mi ds-idrepo-0 8159m 5593Mi ds-idrepo-1 4253m 5522Mi ds-idrepo-2 4085m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5176m 4020Mi 02:32:13 DEBUG --- stderr --- 02:32:13 DEBUG 02:32:19 INFO 02:32:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:32:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:32:19 INFO [loop_until]: OK (rc = 0) 02:32:19 DEBUG --- stdout --- 02:32:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4733Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10291m 34% 10340Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14140m 47% 3083Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12053m 40% 10290Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 4300Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12728m 42% 10374Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4406m 27% 6293Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10842m 68% 14385Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11228m 70% 13631Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 4250m 26% 6690Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10928m 68% 14442Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8008m 50% 6378Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5393m 33% 7758Mi 13% 02:32:19 DEBUG --- stderr --- 02:32:19 DEBUG 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "1738.1" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971981" 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "instance": "10.142.0.107:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "instance": "10.142.0.77:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "593847694336" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-idrepo-0" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712971981" 02:33:01 ERROR [http_cmd]: ERROR 02:33:01 ERROR --- status code --- 02:33:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:33:01 ERROR --- http response --- 02:33:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:33:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0.023" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "1744.4333333333334" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "1747.0666666666666" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "instance": "10.142.0.107:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "instance": "10.142.0.77:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "51410895.766666666" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "1746.6333333333332" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "313820.5" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-cts-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-cts-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "instance": "10.142.0.107:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "instance": "10.142.0.77:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "51405044.400000006" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-cts-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0.0007613000000000001" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-idrepo-0" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-cts-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "5608058.053054711" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "269761516.6059438" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "pod": "ds-idrepo-1" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "pod": "ds-idrepo-0" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "instance": "10.142.0.107:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "instance": "10.142.0.77:9100" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "0.18986666666667323" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712971981" 02:33:01 INFO [http_cmd]: http status code OK 02:33:01 DEBUG --- status code --- 02:33:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:01 DEBUG --- http response --- 02:33:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:01 DEBUG ----- output ----- 02:33:01 DEBUG { 02:33:01 DEBUG "status": "success", 02:33:01 DEBUG "data": { 02:33:01 DEBUG "resultType": "vector", 02:33:01 DEBUG "result": [ 02:33:01 DEBUG { 02:33:01 DEBUG "metric": { 02:33:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG [...] 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG 02:33:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:33:01 DEBUG }, 02:33:01 DEBUG "value": [ 02:33:01 DEBUG 1712971981, 02:33:01 DEBUG "5.253451429033435" 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG ] 02:33:01 DEBUG } 02:33:01 DEBUG } 02:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712971982" 02:33:02 INFO [http_cmd]: http status code OK 02:33:02 DEBUG --- status code --- 02:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:02 DEBUG --- http response --- 02:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:02 DEBUG ----- output ----- 02:33:02 DEBUG { 02:33:02 DEBUG "status": "success", 02:33:02 DEBUG "data": { 02:33:02 DEBUG "resultType": "vector", 02:33:02 DEBUG "result": [ 02:33:02 DEBUG { 02:33:02 DEBUG "metric": { 02:33:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG [...] 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 1712971982, 02:33:02 DEBUG "6238941184" 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG } 02:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971982" 02:33:02 INFO [http_cmd]: http status code OK 02:33:02 DEBUG --- status code --- 02:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:02 DEBUG --- http response --- 02:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:33:02 DEBUG ----- output ----- 02:33:02 DEBUG { 02:33:02 DEBUG "status": "success", 02:33:02 DEBUG "data": { 02:33:02 DEBUG "resultType": "vector", 02:33:02 DEBUG "result": [ 02:33:02 DEBUG { 02:33:02 DEBUG "metric": { 02:33:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG [...] 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 1712971982, 02:33:02 DEBUG "0" 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG } 02:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712971982" 02:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971982" 02:33:02 INFO [http_cmd]: http status code OK 02:33:02 DEBUG --- status code --- 02:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:02 DEBUG --- http response --- 02:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:33:02 DEBUG ----- output ----- 02:33:02 DEBUG { 02:33:02 DEBUG "status": "success", 02:33:02 DEBUG "data": { 02:33:02 DEBUG "resultType": "vector", 02:33:02 DEBUG "result": [ 02:33:02 DEBUG { 02:33:02 DEBUG "metric": { 02:33:02 DEBUG "instance": "10.142.0.107:9100" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG [...] 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG "instance": "10.142.0.77:9100" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 1712971982, 02:33:02 DEBUG "0.06533333333333384" 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG } 02:33:02 INFO [http_cmd]: http status code OK 02:33:02 DEBUG --- status code --- 02:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:02 DEBUG --- http response --- 02:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:33:02 DEBUG ----- output ----- 02:33:02 DEBUG { 02:33:02 DEBUG "status": "success", 02:33:02 DEBUG "data": { 02:33:02 DEBUG "resultType": "vector", 02:33:02 DEBUG "result": [ 02:33:02 DEBUG { 02:33:02 DEBUG "metric": { 02:33:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG [...] 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 1712971982, 02:33:02 DEBUG "12.028452651230038" 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG } 02:33:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712971982" 02:33:02 INFO [http_cmd]: http status code OK 02:33:02 DEBUG --- status code --- 02:33:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:33:02 DEBUG --- http response --- 02:33:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:33:02 DEBUG ----- output ----- 02:33:02 DEBUG { 02:33:02 DEBUG "status": "success", 02:33:02 DEBUG "data": { 02:33:02 DEBUG "resultType": "vector", 02:33:02 DEBUG "result": [ 02:33:02 DEBUG { 02:33:02 DEBUG "metric": { 02:33:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG [...] 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG 02:33:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:33:02 DEBUG }, 02:33:02 DEBUG "value": [ 02:33:02 DEBUG 1712971982, 02:33:02 DEBUG "0" 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG ] 02:33:02 DEBUG } 02:33:02 DEBUG } 02:33:13 INFO 02:33:13 INFO [loop_until]: kubectl --namespace=xlou top pods 02:33:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:33:14 INFO [loop_until]: OK (rc = 0) 02:33:14 DEBUG --- stdout --- 02:33:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12090m 8541Mi am-6d96779b5d-qbg7l 9786m 8466Mi am-6d96779b5d-zn69c 11727m 8488Mi ds-cts-0 10809m 13486Mi ds-cts-1 11053m 12906Mi ds-cts-2 10543m 13499Mi ds-idrepo-0 8009m 5593Mi ds-idrepo-1 4271m 5522Mi ds-idrepo-2 4239m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5238m 4020Mi 02:33:14 DEBUG --- stderr --- 02:33:14 DEBUG 02:33:19 INFO 02:33:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:33:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:33:19 INFO [loop_until]: OK (rc = 0) 02:33:19 DEBUG --- stdout --- 02:33:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 4727Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1853Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10381m 34% 10345Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14166m 47% 3098Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11661m 39% 10287Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4326Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12683m 42% 10396Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4424m 27% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10693m 67% 14380Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11015m 69% 14270Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4283m 26% 6691Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10892m 68% 14364Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8262m 51% 6376Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5391m 33% 7765Mi 13% 02:33:19 DEBUG --- stderr --- 02:33:19 DEBUG 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "1745.4333333333334" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972041" 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "instance": "10.142.0.107:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "instance": "10.142.0.77:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "596405160960" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-idrepo-0" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972041" 02:34:01 ERROR [http_cmd]: ERROR 02:34:01 ERROR --- status code --- 02:34:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:34:01 ERROR --- http response --- 02:34:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:34:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0.023" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "1746.1" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "1746.3" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "314255.5" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "instance": "10.142.0.107:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "instance": "10.142.0.77:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "51394302.86666667" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "1746.3" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-cts-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-cts-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "instance": "10.142.0.107:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "instance": "10.142.0.77:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "51389521.56666666" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-cts-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0.0005666666666666667" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-idrepo-0" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-cts-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "41755155.58611704" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "592.9571857696066" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972041" 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "pod": "ds-idrepo-1" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "pod": "ds-idrepo-0" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "instance": "10.142.0.107:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "instance": "10.142.0.77:9100" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "0.17793333333333078" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972041" 02:34:01 INFO [http_cmd]: http status code OK 02:34:01 DEBUG --- status code --- 02:34:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:01 DEBUG --- http response --- 02:34:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:01 DEBUG ----- output ----- 02:34:01 DEBUG { 02:34:01 DEBUG "status": "success", 02:34:01 DEBUG "data": { 02:34:01 DEBUG "resultType": "vector", 02:34:01 DEBUG "result": [ 02:34:01 DEBUG { 02:34:01 DEBUG "metric": { 02:34:01 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG [...] 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG 02:34:01 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:34:01 DEBUG }, 02:34:01 DEBUG "value": [ 02:34:01 DEBUG 1712972041, 02:34:01 DEBUG "5.256868044279614" 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG ] 02:34:01 DEBUG } 02:34:01 DEBUG } 02:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972042" 02:34:02 INFO [http_cmd]: http status code OK 02:34:02 DEBUG --- status code --- 02:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:02 DEBUG --- http response --- 02:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:02 DEBUG ----- output ----- 02:34:02 DEBUG { 02:34:02 DEBUG "status": "success", 02:34:02 DEBUG "data": { 02:34:02 DEBUG "resultType": "vector", 02:34:02 DEBUG "result": [ 02:34:02 DEBUG { 02:34:02 DEBUG "metric": { 02:34:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG [...] 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 1712972042, 02:34:02 DEBUG "6238871552" 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG } 02:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972042" 02:34:02 INFO [http_cmd]: http status code OK 02:34:02 DEBUG --- status code --- 02:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:02 DEBUG --- http response --- 02:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:34:02 DEBUG ----- output ----- 02:34:02 DEBUG { 02:34:02 DEBUG "status": "success", 02:34:02 DEBUG "data": { 02:34:02 DEBUG "resultType": "vector", 02:34:02 DEBUG "result": [ 02:34:02 DEBUG { 02:34:02 DEBUG "metric": { 02:34:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG [...] 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 1712972042, 02:34:02 DEBUG "0" 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG } 02:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972042" 02:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972042" 02:34:02 INFO [http_cmd]: http status code OK 02:34:02 DEBUG --- status code --- 02:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:02 DEBUG --- http response --- 02:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:34:02 DEBUG ----- output ----- 02:34:02 DEBUG { 02:34:02 DEBUG "status": "success", 02:34:02 DEBUG "data": { 02:34:02 DEBUG "resultType": "vector", 02:34:02 DEBUG "result": [ 02:34:02 DEBUG { 02:34:02 DEBUG "metric": { 02:34:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG [...] 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 1712972042, 02:34:02 DEBUG "0.0032881546089538844" 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG } 02:34:02 INFO [http_cmd]: http status code OK 02:34:02 DEBUG --- status code --- 02:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:02 DEBUG --- http response --- 02:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:34:02 DEBUG ----- output ----- 02:34:02 DEBUG { 02:34:02 DEBUG "status": "success", 02:34:02 DEBUG "data": { 02:34:02 DEBUG "resultType": "vector", 02:34:02 DEBUG "result": [ 02:34:02 DEBUG { 02:34:02 DEBUG "metric": { 02:34:02 DEBUG "instance": "10.142.0.107:9100" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG [...] 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG "instance": "10.142.0.77:9100" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 1712972042, 02:34:02 DEBUG "0.05433333333333247" 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG } 02:34:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972042" 02:34:02 INFO [http_cmd]: http status code OK 02:34:02 DEBUG --- status code --- 02:34:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:34:02 DEBUG --- http response --- 02:34:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:34:02 DEBUG ----- output ----- 02:34:02 DEBUG { 02:34:02 DEBUG "status": "success", 02:34:02 DEBUG "data": { 02:34:02 DEBUG "resultType": "vector", 02:34:02 DEBUG "result": [ 02:34:02 DEBUG { 02:34:02 DEBUG "metric": { 02:34:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG [...] 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG 02:34:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:34:02 DEBUG }, 02:34:02 DEBUG "value": [ 02:34:02 DEBUG 1712972042, 02:34:02 DEBUG "0" 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG ] 02:34:02 DEBUG } 02:34:02 DEBUG } 02:34:14 INFO 02:34:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:34:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:34:14 INFO [loop_until]: OK (rc = 0) 02:34:14 DEBUG --- stdout --- 02:34:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12109m 8553Mi am-6d96779b5d-qbg7l 9559m 8468Mi am-6d96779b5d-zn69c 11850m 8480Mi ds-cts-0 10652m 13488Mi ds-cts-1 10860m 13501Mi ds-cts-2 10770m 13498Mi ds-idrepo-0 8180m 5594Mi ds-idrepo-1 4238m 5522Mi ds-idrepo-2 4080m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 15m 1095Mi idm-b85f9d978-dz57p 7m 1116Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5413m 4024Mi 02:34:14 DEBUG --- stderr --- 02:34:14 DEBUG 02:34:19 INFO 02:34:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:34:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:34:19 INFO [loop_until]: OK (rc = 0) 02:34:19 DEBUG --- stdout --- 02:34:19 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4742Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1864Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10134m 33% 10349Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14134m 47% 3078Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12063m 40% 10281Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 148m 0% 4318Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12640m 42% 10391Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4286m 26% 6293Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10770m 67% 14372Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11297m 71% 14359Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4238m 26% 6694Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10875m 68% 14366Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8246m 51% 6377Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5401m 33% 7765Mi 13% 02:34:19 DEBUG --- stderr --- 02:34:19 DEBUG 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "1742.6333333333332" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 ERROR [http_cmd]: ERROR 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG --- status code --- 02:35:01 ERROR --- status code --- 02:35:01 DEBUG { 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG --- http response --- 02:35:01 ERROR --- http response --- 02:35:01 DEBUG "data": { 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG "result": [ 02:35:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:35:01 DEBUG { 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "data": { 02:35:01 DEBUG "instance": "10.142.0.107:9100" 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG }, 02:35:01 DEBUG "result": [ 02:35:01 DEBUG "value": [ 02:35:01 DEBUG { 02:35:01 DEBUG 02:35:01 DEBUG "metric": { 02:35:01 DEBUG 02:35:01 DEBUG "pod": "ds-idrepo-1" 02:35:01 DEBUG 02:35:01 DEBUG }, 02:35:01 DEBUG [...] 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "instance": "10.142.0.77:9100" 02:35:01 DEBUG [...] 02:35:01 DEBUG }, 02:35:01 DEBUG 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG 02:35:01 DEBUG "599169858560" 02:35:01 DEBUG "pod": "ds-idrepo-0" 02:35:01 DEBUG ] 02:35:01 DEBUG }, 02:35:01 DEBUG } 02:35:01 DEBUG "value": [ 02:35:01 DEBUG ] 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG } 02:35:01 DEBUG "0" 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "1739.7666666666667" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "0.023" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "1738.5666666666666" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "314508" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "instance": "10.142.0.107:9100" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "instance": "10.142.0.77:9100" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "51176241.96666667" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "1738.3999999999999" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "ds-idrepo-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "ds-cts-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "0.00003333333333333336" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "ds-idrepo-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "ds-cts-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "0" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "ds-idrepo-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "ds-cts-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "22.466666666666665" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "instance": "10.142.0.107:9100" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "instance": "10.142.0.77:9100" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "51170774.46666667" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "ds-idrepo-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "ds-cts-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "0" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "ds-idrepo-1" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "ds-idrepo-0" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "0" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "9894.980748553267" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: http status code OK 02:35:01 DEBUG --- status code --- 02:35:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:01 DEBUG --- http response --- 02:35:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:35:01 DEBUG ----- output ----- 02:35:01 DEBUG { 02:35:01 DEBUG "status": "success", 02:35:01 DEBUG "data": { 02:35:01 DEBUG "resultType": "vector", 02:35:01 DEBUG "result": [ 02:35:01 DEBUG { 02:35:01 DEBUG "metric": { 02:35:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG [...] 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG 02:35:01 DEBUG "pod": "idm-b85f9d978-b559t" 02:35:01 DEBUG }, 02:35:01 DEBUG "value": [ 02:35:01 DEBUG 1712972101, 02:35:01 DEBUG "1134.0172918615417" 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG ] 02:35:01 DEBUG } 02:35:01 DEBUG } 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972101" 02:35:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972101" 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "pod": "ds-idrepo-1" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "pod": "ds-idrepo-0" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972101, 02:35:02 DEBUG "0" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972101, 02:35:02 DEBUG "5.2465202660990595" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972102" 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "instance": "10.142.0.107:9100" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "instance": "10.142.0.77:9100" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972102, 02:35:02 DEBUG "0.17373333333333912" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972102" 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972102, 02:35:02 DEBUG "6238826496" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972102" 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "pod": "ds-idrepo-1" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972102, 02:35:02 DEBUG "0" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972102" 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "instance": "10.142.0.107:9100" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "instance": "10.142.0.77:9100" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972102, 02:35:02 DEBUG "0.04933333333333465" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972102" 02:35:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972102" 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "pod": "ds-idrepo-1" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972102, 02:35:02 DEBUG "4.257159471688117" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:02 INFO [http_cmd]: http status code OK 02:35:02 DEBUG --- status code --- 02:35:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:35:02 DEBUG --- http response --- 02:35:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:35:02 DEBUG ----- output ----- 02:35:02 DEBUG { 02:35:02 DEBUG "status": "success", 02:35:02 DEBUG "data": { 02:35:02 DEBUG "resultType": "vector", 02:35:02 DEBUG "result": [ 02:35:02 DEBUG { 02:35:02 DEBUG "metric": { 02:35:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG [...] 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG 02:35:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:35:02 DEBUG }, 02:35:02 DEBUG "value": [ 02:35:02 DEBUG 1712972102, 02:35:02 DEBUG "0" 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG ] 02:35:02 DEBUG } 02:35:02 DEBUG } 02:35:14 INFO 02:35:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:35:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:35:14 INFO [loop_until]: OK (rc = 0) 02:35:14 DEBUG --- stdout --- 02:35:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11970m 8548Mi am-6d96779b5d-qbg7l 9680m 8467Mi am-6d96779b5d-zn69c 11648m 8479Mi ds-cts-0 10651m 13533Mi ds-cts-1 11337m 13502Mi ds-cts-2 10882m 13500Mi ds-idrepo-0 8106m 5593Mi ds-idrepo-1 4208m 5522Mi ds-idrepo-2 4181m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5092m 4024Mi 02:35:14 DEBUG --- stderr --- 02:35:14 DEBUG 02:35:19 INFO 02:35:19 INFO [loop_until]: kubectl --namespace=xlou top node 02:35:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:35:20 INFO [loop_until]: OK (rc = 0) 02:35:20 DEBUG --- stdout --- 02:35:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4731Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1869Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9974m 33% 10342Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14031m 46% 3102Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12021m 40% 10281Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4312Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12818m 42% 10395Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4407m 27% 6299Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10935m 68% 14372Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11285m 71% 14361Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4239m 26% 6693Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 11013m 69% 14370Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8287m 52% 6375Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5371m 33% 7763Mi 13% 02:35:20 DEBUG --- stderr --- 02:35:20 DEBUG 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "1759.5" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "ds-idrepo-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "ds-idrepo-0" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "0" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "instance": "10.142.0.107:9100" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "instance": "10.142.0.77:9100" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "601974057984" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972161" 02:36:01 ERROR [http_cmd]: ERROR 02:36:01 ERROR --- status code --- 02:36:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:36:01 ERROR --- http response --- 02:36:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:36:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "0.023" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "1756.4333333333334" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "1763.3999999999999" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972161" 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "instance": "10.142.0.107:9100" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "instance": "10.142.0.77:9100" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "51605180.93333334" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "314517.5" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "ds-idrepo-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "ds-cts-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "0" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "1763.8" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "ds-idrepo-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "ds-cts-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "0.0029121527777777774" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "ds-idrepo-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "ds-cts-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "28.9" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "instance": "10.142.0.107:9100" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "instance": "10.142.0.77:9100" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "51599764.6" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "ds-idrepo-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "ds-idrepo-0" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "0" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "ds-idrepo-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "ds-cts-1" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "0" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972161" 02:36:01 INFO [http_cmd]: http status code OK 02:36:01 DEBUG --- status code --- 02:36:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:01 DEBUG --- http response --- 02:36:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:36:01 DEBUG ----- output ----- 02:36:01 DEBUG { 02:36:01 DEBUG "status": "success", 02:36:01 DEBUG "data": { 02:36:01 DEBUG "resultType": "vector", 02:36:01 DEBUG "result": [ 02:36:01 DEBUG { 02:36:01 DEBUG "metric": { 02:36:01 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG [...] 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG 02:36:01 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:36:01 DEBUG }, 02:36:01 DEBUG "value": [ 02:36:01 DEBUG 1712972161, 02:36:01 DEBUG "7372.863128043894" 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG ] 02:36:01 DEBUG } 02:36:01 DEBUG } 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972162" 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "0" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972162" 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972162" 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972162" 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "pod": "ds-idrepo-1" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "pod": "ds-idrepo-0" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "0" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "instance": "10.142.0.107:9100" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "instance": "10.142.0.77:9100" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "0.16983333333334136" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "5.252293972744006" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972162" 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "6238175232" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972162" 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "0" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972162" 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "instance": "10.142.0.107:9100" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972162" 02:36:02 DEBUG "instance": "10.142.0.77:9100" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "0.04733333333333434" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972162" 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "0" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:02 INFO [http_cmd]: http status code OK 02:36:02 DEBUG --- status code --- 02:36:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:36:02 DEBUG --- http response --- 02:36:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:36:02 DEBUG ----- output ----- 02:36:02 DEBUG { 02:36:02 DEBUG "status": "success", 02:36:02 DEBUG "data": { 02:36:02 DEBUG "resultType": "vector", 02:36:02 DEBUG "result": [ 02:36:02 DEBUG { 02:36:02 DEBUG "metric": { 02:36:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG [...] 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG 02:36:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:36:02 DEBUG }, 02:36:02 DEBUG "value": [ 02:36:02 DEBUG 1712972162, 02:36:02 DEBUG "7.981607578488431" 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG ] 02:36:02 DEBUG } 02:36:02 DEBUG } 02:36:14 INFO 02:36:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:36:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:36:14 INFO [loop_until]: OK (rc = 0) 02:36:14 DEBUG --- stdout --- 02:36:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11850m 8550Mi am-6d96779b5d-qbg7l 9971m 8471Mi am-6d96779b5d-zn69c 11950m 8478Mi ds-cts-0 10845m 13494Mi ds-cts-1 10975m 13582Mi ds-cts-2 10710m 13500Mi ds-idrepo-0 8086m 5593Mi ds-idrepo-1 4381m 5522Mi ds-idrepo-2 4127m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 7m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5317m 4024Mi 02:36:14 DEBUG --- stderr --- 02:36:14 DEBUG 02:36:20 INFO 02:36:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:36:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:36:20 INFO [loop_until]: OK (rc = 0) 02:36:20 DEBUG --- stdout --- 02:36:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 88m 0% 4730Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1870Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10300m 34% 10346Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14084m 47% 3093Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12070m 40% 10277Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 4324Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12438m 41% 10392Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4367m 27% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10554m 66% 14376Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11535m 72% 14369Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4252m 26% 6691Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10657m 67% 14525Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8248m 51% 6377Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5475m 34% 7764Mi 13% 02:36:20 DEBUG --- stderr --- 02:36:20 DEBUG 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "1746.2" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "ds-idrepo-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "ds-idrepo-0" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "0" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "instance": "10.142.0.107:9100" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "instance": "10.142.0.77:9100" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "604769793024" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972221" 02:37:01 ERROR [http_cmd]: ERROR 02:37:01 ERROR --- status code --- 02:37:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:37:01 ERROR --- http response --- 02:37:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:37:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "1744.3333333333333" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "0.023" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "1745.2" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972221" 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "instance": "10.142.0.107:9100" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "instance": "10.142.0.77:9100" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "51138363.53333333" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "314941" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "ds-idrepo-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "ds-cts-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "0" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "1745.2333333333333" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "ds-idrepo-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "ds-cts-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "0.0002548027777777777" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "ds-idrepo-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "ds-cts-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "28.566666666666666" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "instance": "10.142.0.107:9100" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "instance": "10.142.0.77:9100" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "51134335.46666667" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "ds-idrepo-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "ds-idrepo-0" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "0" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972221" 02:37:01 INFO [http_cmd]: http status code OK 02:37:01 DEBUG --- status code --- 02:37:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:01 DEBUG --- http response --- 02:37:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:37:01 DEBUG ----- output ----- 02:37:01 DEBUG { 02:37:01 DEBUG "status": "success", 02:37:01 DEBUG "data": { 02:37:01 DEBUG "resultType": "vector", 02:37:01 DEBUG "result": [ 02:37:01 DEBUG { 02:37:01 DEBUG "metric": { 02:37:01 DEBUG "pod": "ds-idrepo-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG [...] 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG 02:37:01 DEBUG "pod": "ds-cts-1" 02:37:01 DEBUG }, 02:37:01 DEBUG "value": [ 02:37:01 DEBUG 1712972221, 02:37:01 DEBUG "0" 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG ] 02:37:01 DEBUG } 02:37:01 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "pod": "idm-b85f9d978-dz57p" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "7398.680281638221" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "pod": "idm-b85f9d978-dz57p" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "301.9869502709478" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "instance": "10.142.0.107:9100" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "instance": "10.142.0.77:9100" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "0.17043333333332858" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "pod": "ds-idrepo-1" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "pod": "ds-idrepo-0" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "0" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "5.289931358844779" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "6237626368" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "pod": "idm-b85f9d978-dz57p" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "0" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972222" 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "pod": "idm-b85f9d978-dz57p" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "0.002916208943119064" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "0" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972222" 02:37:02 INFO [http_cmd]: http status code OK 02:37:02 DEBUG --- status code --- 02:37:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:37:02 DEBUG --- http response --- 02:37:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:37:02 DEBUG ----- output ----- 02:37:02 DEBUG { 02:37:02 DEBUG "status": "success", 02:37:02 DEBUG "data": { 02:37:02 DEBUG "resultType": "vector", 02:37:02 DEBUG "result": [ 02:37:02 DEBUG { 02:37:02 DEBUG "metric": { 02:37:02 DEBUG "instance": "10.142.0.107:9100" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG [...] 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG 02:37:02 DEBUG "instance": "10.142.0.77:9100" 02:37:02 DEBUG }, 02:37:02 DEBUG "value": [ 02:37:02 DEBUG 1712972222, 02:37:02 DEBUG "0.04666666666666663" 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG ] 02:37:02 DEBUG } 02:37:02 DEBUG } 02:37:14 INFO 02:37:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:37:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:37:14 INFO [loop_until]: OK (rc = 0) 02:37:14 DEBUG --- stdout --- 02:37:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12281m 8554Mi am-6d96779b5d-qbg7l 10081m 8470Mi am-6d96779b5d-zn69c 11674m 8475Mi ds-cts-0 10835m 13493Mi ds-cts-1 10944m 13503Mi ds-cts-2 10468m 13499Mi ds-idrepo-0 8162m 5593Mi ds-idrepo-1 4367m 5522Mi ds-idrepo-2 4081m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 10m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5247m 4024Mi 02:37:14 DEBUG --- stderr --- 02:37:14 DEBUG 02:37:20 INFO 02:37:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:37:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:37:20 INFO [loop_until]: OK (rc = 0) 02:37:20 DEBUG --- stdout --- 02:37:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4728Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1865Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10254m 34% 10345Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14156m 47% 3094Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12222m 40% 10276Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4312Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12805m 42% 10398Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4382m 27% 6297Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10881m 68% 14381Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11279m 70% 14359Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4251m 26% 6692Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10943m 68% 14486Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8272m 52% 6379Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5400m 33% 7759Mi 13% 02:37:20 DEBUG --- stderr --- 02:37:20 DEBUG 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "1717.9333333333334" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "ds-idrepo-1" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "ds-idrepo-0" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "0" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "instance": "10.142.0.107:9100" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "instance": "10.142.0.77:9100" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "607371090944" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 ERROR [http_cmd]: ERROR 02:38:01 ERROR --- status code --- 02:38:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:38:01 ERROR --- http response --- 02:38:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:38:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "0.023" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "1734" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "1722.9333333333334" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972281" 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "instance": "10.142.0.107:9100" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "instance": "10.142.0.77:9100" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "50934603.36666667" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "314463.5" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "ds-idrepo-1" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "ds-cts-1" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "0" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "1723.2333333333333" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972281" 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "ds-idrepo-1" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "ds-idrepo-0" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "0" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:01 INFO [http_cmd]: http status code OK 02:38:01 DEBUG --- status code --- 02:38:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:01 DEBUG --- http response --- 02:38:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:38:01 DEBUG ----- output ----- 02:38:01 DEBUG { 02:38:01 DEBUG "status": "success", 02:38:01 DEBUG "data": { 02:38:01 DEBUG "resultType": "vector", 02:38:01 DEBUG "result": [ 02:38:01 DEBUG { 02:38:01 DEBUG "metric": { 02:38:01 DEBUG "pod": "ds-idrepo-1" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG [...] 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG 02:38:01 DEBUG "pod": "ds-cts-1" 02:38:01 DEBUG }, 02:38:01 DEBUG "value": [ 02:38:01 DEBUG 1712972281, 02:38:01 DEBUG "0.0004649722222222223" 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG ] 02:38:01 DEBUG } 02:38:01 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-cts-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "29.099999999999998" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "instance": "10.142.0.107:9100" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "instance": "10.142.0.77:9100" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "50928852.83333333" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-cts-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "0" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "165617295.0964975" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972282" 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "3044.917798427448" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-idrepo-0" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "0" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "instance": "10.142.0.107:9100" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "instance": "10.142.0.77:9100" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "0.16850000000000923" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "5.25049210849303" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "6237392896" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "0" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "ds-idrepo-1" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "4.2832340516941025" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "0" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972282" 02:38:02 INFO [http_cmd]: http status code OK 02:38:02 DEBUG --- status code --- 02:38:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:38:02 DEBUG --- http response --- 02:38:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:38:02 DEBUG ----- output ----- 02:38:02 DEBUG { 02:38:02 DEBUG "status": "success", 02:38:02 DEBUG "data": { 02:38:02 DEBUG "resultType": "vector", 02:38:02 DEBUG "result": [ 02:38:02 DEBUG { 02:38:02 DEBUG "metric": { 02:38:02 DEBUG "instance": "10.142.0.107:9100" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG [...] 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG 02:38:02 DEBUG "instance": "10.142.0.77:9100" 02:38:02 DEBUG }, 02:38:02 DEBUG "value": [ 02:38:02 DEBUG 1712972282, 02:38:02 DEBUG "0.0480000000000011" 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG ] 02:38:02 DEBUG } 02:38:02 DEBUG } 02:38:14 INFO 02:38:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:38:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:38:14 INFO [loop_until]: OK (rc = 0) 02:38:14 DEBUG --- stdout --- 02:38:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12090m 8555Mi am-6d96779b5d-qbg7l 10053m 8472Mi am-6d96779b5d-zn69c 11887m 8476Mi ds-cts-0 10698m 13493Mi ds-cts-1 11320m 13551Mi ds-cts-2 10487m 13499Mi ds-idrepo-0 8031m 5594Mi ds-idrepo-1 4247m 5522Mi ds-idrepo-2 4029m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1095Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5347m 4025Mi 02:38:14 DEBUG --- stderr --- 02:38:14 DEBUG 02:38:20 INFO 02:38:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:38:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:38:20 INFO [loop_until]: OK (rc = 0) 02:38:20 DEBUG --- stdout --- 02:38:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4734Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1853Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10252m 34% 10344Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14123m 47% 3091Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12144m 40% 10289Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 4304Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12735m 42% 10402Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4407m 27% 6297Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10816m 68% 14388Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10905m 68% 14365Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4111m 25% 6689Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10933m 68% 14379Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8286m 52% 6375Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5396m 33% 7760Mi 13% 02:38:20 DEBUG --- stderr --- 02:38:20 DEBUG 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "1764" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "instance": "10.142.0.107:9100" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "instance": "10.142.0.77:9100" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "609640760320" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "pod": "ds-idrepo-1" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "pod": "ds-idrepo-0" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "0" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 ERROR [http_cmd]: ERROR 02:39:01 ERROR --- status code --- 02:39:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:39:01 ERROR --- http response --- 02:39:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:39:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "0.023" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "1741.5666666666666" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "1746.9666666666667" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972341" 02:39:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972341" 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "instance": "10.142.0.107:9100" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "instance": "10.142.0.77:9100" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "51419154.33333333" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:01 INFO [http_cmd]: http status code OK 02:39:01 DEBUG --- status code --- 02:39:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:01 DEBUG --- http response --- 02:39:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:01 DEBUG ----- output ----- 02:39:01 DEBUG { 02:39:01 DEBUG "status": "success", 02:39:01 DEBUG "data": { 02:39:01 DEBUG "resultType": "vector", 02:39:01 DEBUG "result": [ 02:39:01 DEBUG { 02:39:01 DEBUG "metric": { 02:39:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG [...] 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG 02:39:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:01 DEBUG }, 02:39:01 DEBUG "value": [ 02:39:01 DEBUG 1712972341, 02:39:01 DEBUG "314456.5" 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG ] 02:39:01 DEBUG } 02:39:01 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "ds-idrepo-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "ds-cts-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "1747.8" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "ds-idrepo-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "ds-idrepo-0" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "ds-idrepo-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "ds-cts-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0.0007141444444444446" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "ds-idrepo-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "ds-cts-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "28.2" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "instance": "10.142.0.107:9100" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "instance": "10.142.0.77:9100" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "51413588.3" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "ds-idrepo-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "ds-cts-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "instance": "10.142.0.107:9100" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "instance": "10.142.0.77:9100" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0.163633333333343" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "ds-idrepo-1" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "ds-idrepo-0" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "585.3309992497587" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "5.252913664718071" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "6238593024" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972342" 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972342" 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "instance": "10.142.0.107:9100" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "instance": "10.142.0.77:9100" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0.04800000000000182" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:02 INFO [http_cmd]: http status code OK 02:39:02 DEBUG --- status code --- 02:39:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:39:02 DEBUG --- http response --- 02:39:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:39:02 DEBUG ----- output ----- 02:39:02 DEBUG { 02:39:02 DEBUG "status": "success", 02:39:02 DEBUG "data": { 02:39:02 DEBUG "resultType": "vector", 02:39:02 DEBUG "result": [ 02:39:02 DEBUG { 02:39:02 DEBUG "metric": { 02:39:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG [...] 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG 02:39:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:39:02 DEBUG }, 02:39:02 DEBUG "value": [ 02:39:02 DEBUG 1712972342, 02:39:02 DEBUG "0" 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG ] 02:39:02 DEBUG } 02:39:02 DEBUG } 02:39:14 INFO 02:39:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:39:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:39:14 INFO [loop_until]: OK (rc = 0) 02:39:14 DEBUG --- stdout --- 02:39:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12190m 8560Mi am-6d96779b5d-qbg7l 9800m 8470Mi am-6d96779b5d-zn69c 11682m 8484Mi ds-cts-0 10854m 13502Mi ds-cts-1 11117m 13048Mi ds-cts-2 10529m 13500Mi ds-idrepo-0 8063m 5594Mi ds-idrepo-1 4312m 5522Mi ds-idrepo-2 4251m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5081m 4024Mi 02:39:14 DEBUG --- stderr --- 02:39:14 DEBUG 02:39:20 INFO 02:39:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:39:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:39:20 INFO [loop_until]: OK (rc = 0) 02:39:20 DEBUG --- stdout --- 02:39:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4740Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1859Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10209m 34% 10351Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14027m 46% 3100Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12034m 40% 10285Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 148m 0% 4338Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12744m 42% 10396Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4258m 26% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10834m 68% 14382Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11165m 70% 14213Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4233m 26% 6691Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10946m 68% 14471Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8258m 51% 6378Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5185m 32% 7761Mi 13% 02:39:20 DEBUG --- stderr --- 02:39:20 DEBUG 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972401" 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972401" 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:01 DEBUG ----- output ----- 02:40:01 DEBUG { 02:40:01 DEBUG "status": "success", 02:40:01 DEBUG "data": { 02:40:01 DEBUG "resultType": "vector", 02:40:01 DEBUG "result": [ 02:40:01 DEBUG { 02:40:01 DEBUG "metric": { 02:40:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG [...] 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 1712972401, 02:40:01 DEBUG "1736.6666666666667" 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG } 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972401" 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:01 DEBUG ----- output ----- 02:40:01 DEBUG { 02:40:01 DEBUG "status": "success", 02:40:01 DEBUG "data": { 02:40:01 DEBUG "resultType": "vector", 02:40:01 DEBUG "result": [ 02:40:01 DEBUG { 02:40:01 DEBUG "metric": { 02:40:01 DEBUG "instance": "10.142.0.107:9100" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG [...] 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG "instance": "10.142.0.77:9100" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 1712972401, 02:40:01 DEBUG "612358368256" 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG } 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972401" 02:40:01 ERROR [http_cmd]: ERROR 02:40:01 ERROR --- status code --- 02:40:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:40:01 ERROR --- http response --- 02:40:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:40:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972401" 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:01 DEBUG ----- output ----- 02:40:01 DEBUG { 02:40:01 DEBUG "status": "success", 02:40:01 DEBUG "data": { 02:40:01 DEBUG "resultType": "vector", 02:40:01 DEBUG "result": [ 02:40:01 DEBUG { 02:40:01 DEBUG "metric": { 02:40:01 DEBUG "pod": "ds-idrepo-1" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG [...] 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG "pod": "ds-idrepo-0" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 1712972401, 02:40:01 DEBUG "0" 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG } 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972401" 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972401" 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:01 DEBUG ----- output ----- 02:40:01 DEBUG { 02:40:01 DEBUG "status": "success", 02:40:01 DEBUG "data": { 02:40:01 DEBUG "resultType": "vector", 02:40:01 DEBUG "result": [ 02:40:01 DEBUG { 02:40:01 DEBUG "metric": { 02:40:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG [...] 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 1712972401, 02:40:01 DEBUG "0.023" 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG } 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:01 DEBUG ----- output ----- 02:40:01 DEBUG { 02:40:01 DEBUG "status": "success", 02:40:01 DEBUG "data": { 02:40:01 DEBUG "resultType": "vector", 02:40:01 DEBUG "result": [ 02:40:01 DEBUG { 02:40:01 DEBUG "metric": { 02:40:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG [...] 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG 02:40:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:01 DEBUG 1712972401, 02:40:01 DEBUG "1749.9666666666667" 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG ] 02:40:01 DEBUG } 02:40:01 DEBUG } 02:40:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972401" 02:40:01 INFO [http_cmd]: http status code OK 02:40:01 DEBUG --- status code --- 02:40:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:01 DEBUG --- http response --- 02:40:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:01 DEBUG ----- output ----- 02:40:01 DEBUG { 02:40:01 DEBUG "status": "success", 02:40:01 DEBUG "data": { 02:40:01 DEBUG "resultType": "vector", 02:40:01 DEBUG "result": [ 02:40:01 DEBUG { 02:40:01 DEBUG "metric": { 02:40:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:01 DEBUG }, 02:40:01 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972401, 02:40:02 DEBUG "1747.8666666666666" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972402" 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "instance": "10.142.0.107:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "instance": "10.142.0.77:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "51263296.733333334" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "313982" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "ds-idrepo-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "ds-cts-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "ds-idrepo-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "ds-cts-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "1746.6333333333332" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "ds-idrepo-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "ds-idrepo-0" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "ds-idrepo-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "ds-cts-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "27.766666666666666" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972402" 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG { 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "ds-idrepo-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "ds-cts-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "instance": "10.142.0.107:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "instance": "10.142.0.77:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "51257371.733333334" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "8221.85164597616" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972402" 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "ds-idrepo-1" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:02 DEBUG 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG 02:40:02 DEBUG { 02:40:02 DEBUG 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "pod": "ds-idrepo-0" 02:40:02 DEBUG "data": { 02:40:02 DEBUG }, 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "value": [ 02:40:02 DEBUG "result": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG { 02:40:02 DEBUG "0" 02:40:02 DEBUG "metric": { 02:40:02 DEBUG ] 02:40:02 DEBUG "instance": "10.142.0.107:9100" 02:40:02 DEBUG } 02:40:02 DEBUG }, 02:40:02 DEBUG ] 02:40:02 DEBUG "value": [ 02:40:02 DEBUG } 02:40:02 DEBUG 02:40:02 DEBUG } 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "instance": "10.142.0.77:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0.1722666666666773" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "505.9601012908406" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "5.2338062457424686" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "6238474240" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "instance": "10.142.0.107:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "instance": "10.142.0.77:9100" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0.05166666666666705" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972402" 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0.0028831758384290887" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:02 INFO [http_cmd]: http status code OK 02:40:02 DEBUG --- status code --- 02:40:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:40:02 DEBUG --- http response --- 02:40:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:40:02 DEBUG ----- output ----- 02:40:02 DEBUG { 02:40:02 DEBUG "status": "success", 02:40:02 DEBUG "data": { 02:40:02 DEBUG "resultType": "vector", 02:40:02 DEBUG "result": [ 02:40:02 DEBUG { 02:40:02 DEBUG "metric": { 02:40:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG [...] 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG 02:40:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:40:02 DEBUG }, 02:40:02 DEBUG "value": [ 02:40:02 DEBUG 1712972402, 02:40:02 DEBUG "0" 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG ] 02:40:02 DEBUG } 02:40:02 DEBUG } 02:40:14 INFO 02:40:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:40:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:40:14 INFO [loop_until]: OK (rc = 0) 02:40:14 DEBUG --- stdout --- 02:40:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11827m 8555Mi am-6d96779b5d-qbg7l 9757m 8471Mi am-6d96779b5d-zn69c 11420m 8478Mi ds-cts-0 10560m 13504Mi ds-cts-1 10722m 13581Mi ds-cts-2 10816m 13500Mi ds-idrepo-0 8205m 5594Mi ds-idrepo-1 4156m 5522Mi ds-idrepo-2 4169m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1095Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5244m 4024Mi 02:40:14 DEBUG --- stderr --- 02:40:14 DEBUG 02:40:20 INFO 02:40:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:40:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:40:20 INFO [loop_until]: OK (rc = 0) 02:40:20 DEBUG --- stdout --- 02:40:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 4745Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10250m 34% 10340Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14059m 47% 3084Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12096m 40% 10279Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 4333Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12783m 42% 10400Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4421m 27% 6297Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10886m 68% 14376Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10905m 68% 14382Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4321m 27% 6700Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10481m 65% 14383Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8363m 52% 6381Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5384m 33% 7762Mi 13% 02:40:20 DEBUG --- stderr --- 02:40:20 DEBUG 02:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972461" 02:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972461" 02:41:01 INFO [http_cmd]: http status code OK 02:41:01 INFO [http_cmd]: http status code OK 02:41:01 DEBUG --- status code --- 02:41:01 DEBUG --- status code --- 02:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:01 DEBUG --- http response --- 02:41:01 DEBUG --- http response --- 02:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:01 DEBUG ----- output ----- 02:41:01 DEBUG { 02:41:01 DEBUG "status": "success", 02:41:01 DEBUG "data": { 02:41:01 DEBUG "resultType": "vector", 02:41:01 DEBUG "result": [ 02:41:01 DEBUG { 02:41:01 DEBUG "metric": { 02:41:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:01 DEBUG }, 02:41:01 DEBUG "value": [ 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG [...] 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:01 DEBUG }, 02:41:01 DEBUG "value": [ 02:41:01 DEBUG 1712972461, 02:41:01 DEBUG "1729.8" 02:41:01 DEBUG ] 02:41:01 DEBUG } 02:41:01 DEBUG ] 02:41:01 DEBUG } 02:41:01 DEBUG } 02:41:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972461" 02:41:01 INFO [http_cmd]: http status code OK 02:41:01 DEBUG --- status code --- 02:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:01 DEBUG --- http response --- 02:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:01 DEBUG ----- output ----- 02:41:01 DEBUG { 02:41:01 DEBUG "status": "success", 02:41:01 DEBUG "data": { 02:41:01 DEBUG "resultType": "vector", 02:41:01 DEBUG "result": [ 02:41:01 DEBUG { 02:41:01 DEBUG "metric": { 02:41:01 DEBUG "instance": "10.142.0.107:9100" 02:41:01 DEBUG }, 02:41:01 DEBUG "value": [ 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG [...] 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG "instance": "10.142.0.77:9100" 02:41:01 DEBUG }, 02:41:01 DEBUG "value": [ 02:41:01 DEBUG 1712972461, 02:41:01 DEBUG "615153331200" 02:41:01 DEBUG ] 02:41:01 DEBUG } 02:41:01 DEBUG ] 02:41:01 DEBUG } 02:41:01 DEBUG } 02:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972461" 02:41:01 INFO [http_cmd]: http status code OK 02:41:01 DEBUG --- status code --- 02:41:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:01 DEBUG --- http response --- 02:41:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:01 DEBUG ----- output ----- 02:41:01 DEBUG { 02:41:01 DEBUG "status": "success", 02:41:01 DEBUG "data": { 02:41:01 DEBUG "resultType": "vector", 02:41:01 DEBUG "result": [ 02:41:01 DEBUG { 02:41:01 DEBUG "metric": { 02:41:01 DEBUG "pod": "ds-idrepo-1" 02:41:01 DEBUG }, 02:41:01 DEBUG "value": [ 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG [...] 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG 02:41:01 DEBUG "pod": "ds-idrepo-0" 02:41:01 DEBUG }, 02:41:01 DEBUG "value": [ 02:41:01 DEBUG 1712972461, 02:41:01 DEBUG "0" 02:41:01 DEBUG ] 02:41:01 DEBUG } 02:41:01 DEBUG ] 02:41:01 DEBUG } 02:41:01 DEBUG } 02:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972461" 02:41:01 ERROR [http_cmd]: ERROR 02:41:01 ERROR --- status code --- 02:41:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:41:01 ERROR --- http response --- 02:41:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:41:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:41:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972461" 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972461, 02:41:02 DEBUG "0.023" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "1736.1333333333334" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "1733.7666666666667" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "314255.5" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "instance": "10.142.0.107:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "instance": "10.142.0.77:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "51129170.766666666" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "ds-idrepo-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "ds-idrepo-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "28.466666666666665" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "ds-idrepo-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0.00010000000000000006" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "1734.2666666666667" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "ds-idrepo-1" 02:41:02 DEBUG "pod": "ds-idrepo-1" 02:41:02 DEBUG }, 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-idrepo-0" 02:41:02 DEBUG "pod": "ds-cts-1" 02:41:02 DEBUG }, 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0" 02:41:02 DEBUG "0" 02:41:02 DEBUG ] 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "instance": "10.142.0.107:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "instance": "10.142.0.77:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "51123276.4" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-2" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "26736.772764020723" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-2" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "44567992.574183315" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "ds-idrepo-1" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-idrepo-0" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "instance": "10.142.0.107:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "instance": "10.142.0.77:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0.17383333333332918" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "5.265070453710682" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "6238203904" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-2" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "29952053.65751415" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "ds-cts-2" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "10.690540009042262" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "instance": "10.142.0.107:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "instance": "10.142.0.77:9100" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0.04833333333333627" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972462" 02:41:02 INFO [http_cmd]: http status code OK 02:41:02 DEBUG --- status code --- 02:41:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:41:02 DEBUG --- http response --- 02:41:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:41:02 DEBUG ----- output ----- 02:41:02 DEBUG { 02:41:02 DEBUG "status": "success", 02:41:02 DEBUG "data": { 02:41:02 DEBUG "resultType": "vector", 02:41:02 DEBUG "result": [ 02:41:02 DEBUG { 02:41:02 DEBUG "metric": { 02:41:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG [...] 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG 02:41:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:41:02 DEBUG }, 02:41:02 DEBUG "value": [ 02:41:02 DEBUG 1712972462, 02:41:02 DEBUG "0" 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG ] 02:41:02 DEBUG } 02:41:02 DEBUG } 02:41:14 INFO 02:41:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:41:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:41:14 INFO [loop_until]: OK (rc = 0) 02:41:14 DEBUG --- stdout --- 02:41:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11695m 8556Mi am-6d96779b5d-qbg7l 9864m 8472Mi am-6d96779b5d-zn69c 11728m 8473Mi ds-cts-0 10544m 13506Mi ds-cts-1 10493m 13513Mi ds-cts-2 10598m 13500Mi ds-idrepo-0 8186m 5593Mi ds-idrepo-1 4369m 5522Mi ds-idrepo-2 4238m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 7m 1102Mi idm-b85f9d978-dz57p 6m 1115Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5272m 4025Mi 02:41:14 DEBUG --- stderr --- 02:41:14 DEBUG 02:41:20 INFO 02:41:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:41:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:41:20 INFO [loop_until]: OK (rc = 0) 02:41:20 DEBUG --- stdout --- 02:41:20 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4735Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1858Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10262m 34% 10353Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14086m 47% 3082Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11730m 39% 10282Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 132m 0% 4334Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12885m 43% 10395Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4343m 27% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10455m 65% 14384Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11497m 72% 14415Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4245m 26% 6691Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10908m 68% 14403Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8251m 51% 6383Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5458m 34% 7863Mi 13% 02:41:20 DEBUG --- stderr --- 02:41:20 DEBUG 02:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972521" 02:42:01 INFO [http_cmd]: http status code OK 02:42:01 DEBUG --- status code --- 02:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:01 DEBUG --- http response --- 02:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:01 DEBUG ----- output ----- 02:42:01 DEBUG { 02:42:01 DEBUG "status": "success", 02:42:01 DEBUG "data": { 02:42:01 DEBUG "resultType": "vector", 02:42:01 DEBUG "result": [ 02:42:01 DEBUG { 02:42:01 DEBUG "metric": { 02:42:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:01 DEBUG }, 02:42:01 DEBUG "value": [ 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG [...] 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:01 DEBUG }, 02:42:01 DEBUG "value": [ 02:42:01 DEBUG 1712972521, 02:42:01 DEBUG "1743.6666666666667" 02:42:01 DEBUG ] 02:42:01 DEBUG } 02:42:01 DEBUG ] 02:42:01 DEBUG } 02:42:01 DEBUG } 02:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972521" 02:42:01 INFO [http_cmd]: http status code OK 02:42:01 DEBUG --- status code --- 02:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:01 DEBUG --- http response --- 02:42:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972521" 02:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972521" 02:42:01 INFO [http_cmd]: http status code OK 02:42:01 DEBUG --- status code --- 02:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:01 DEBUG --- http response --- 02:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:01 DEBUG ----- output ----- 02:42:01 DEBUG { 02:42:01 DEBUG "status": "success", 02:42:01 DEBUG "data": { 02:42:01 DEBUG "resultType": "vector", 02:42:01 DEBUG "result": [ 02:42:01 DEBUG { 02:42:01 DEBUG "metric": { 02:42:01 DEBUG "pod": "ds-idrepo-1" 02:42:01 DEBUG }, 02:42:01 DEBUG "value": [ 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG [...] 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG "pod": "ds-idrepo-0" 02:42:01 DEBUG }, 02:42:01 DEBUG "value": [ 02:42:01 DEBUG 1712972521, 02:42:01 DEBUG "0" 02:42:01 DEBUG ] 02:42:01 DEBUG } 02:42:01 DEBUG ] 02:42:01 DEBUG } 02:42:01 DEBUG } 02:42:01 INFO [http_cmd]: http status code OK 02:42:01 DEBUG --- status code --- 02:42:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:01 DEBUG --- http response --- 02:42:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:01 DEBUG ----- output ----- 02:42:01 DEBUG { 02:42:01 DEBUG "status": "success", 02:42:01 DEBUG "data": { 02:42:01 DEBUG "resultType": "vector", 02:42:01 DEBUG "result": [ 02:42:01 DEBUG { 02:42:01 DEBUG "metric": { 02:42:01 DEBUG "instance": "10.142.0.107:9100" 02:42:01 DEBUG }, 02:42:01 DEBUG "value": [ 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG [...] 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG 02:42:01 DEBUG "instance": "10.142.0.77:9100" 02:42:01 DEBUG }, 02:42:01 DEBUG "value": [ 02:42:01 DEBUG 1712972521, 02:42:01 DEBUG "617931957248" 02:42:01 DEBUG ] 02:42:01 DEBUG } 02:42:01 DEBUG ] 02:42:01 DEBUG } 02:42:01 DEBUG } 02:42:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972521" 02:42:01 ERROR [http_cmd]: ERROR 02:42:01 ERROR --- status code --- 02:42:01 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:42:01 ERROR --- http response --- 02:42:01 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:42:01 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0.023" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "1746.8333333333333" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "1746.3999999999999" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "instance": "10.142.0.107:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "instance": "10.142.0.77:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "51265690.4" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "314093.5" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "ds-idrepo-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "ds-cts-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG "0" 02:42:02 DEBUG { 02:42:02 DEBUG ] 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG } 02:42:02 DEBUG "data": { 02:42:02 DEBUG ] 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG } 02:42:02 DEBUG "result": [ 02:42:02 DEBUG } 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "ds-idrepo-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "ds-cts-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "28.866666666666667" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "1745.2666666666667" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "ds-idrepo-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "ds-cts-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0.0004602833333333334" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "ds-idrepo-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "ds-cts-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "ds-idrepo-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "ds-idrepo-0" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "instance": "10.142.0.107:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "instance": "10.142.0.77:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "51261867.03333333" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "35874503.327278346" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "ds-idrepo-1" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "ds-idrepo-0" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "instance": "10.142.0.107:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "instance": "10.142.0.77:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0.17896666666667424" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "5.326353080141277" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "6238437376" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "instance": "10.142.0.107:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "instance": "10.142.0.77:9100" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0.05366666666666735" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972522" 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0.0013770683039854088" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:02 INFO [http_cmd]: http status code OK 02:42:02 DEBUG --- status code --- 02:42:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:42:02 DEBUG --- http response --- 02:42:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:42:02 DEBUG ----- output ----- 02:42:02 DEBUG { 02:42:02 DEBUG "status": "success", 02:42:02 DEBUG "data": { 02:42:02 DEBUG "resultType": "vector", 02:42:02 DEBUG "result": [ 02:42:02 DEBUG { 02:42:02 DEBUG "metric": { 02:42:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG [...] 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG 02:42:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:42:02 DEBUG }, 02:42:02 DEBUG "value": [ 02:42:02 DEBUG 1712972522, 02:42:02 DEBUG "0" 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG ] 02:42:02 DEBUG } 02:42:02 DEBUG } 02:42:14 INFO 02:42:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:42:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:42:14 INFO [loop_until]: OK (rc = 0) 02:42:14 DEBUG --- stdout --- 02:42:14 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12268m 8556Mi am-6d96779b5d-qbg7l 9725m 8473Mi am-6d96779b5d-zn69c 11663m 8476Mi ds-cts-0 10866m 13507Mi ds-cts-1 11158m 13511Mi ds-cts-2 10527m 13501Mi ds-idrepo-0 8132m 5594Mi ds-idrepo-1 4207m 5522Mi ds-idrepo-2 4159m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1102Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5225m 4122Mi 02:42:14 DEBUG --- stderr --- 02:42:14 DEBUG 02:42:20 INFO 02:42:20 INFO [loop_until]: kubectl --namespace=xlou top node 02:42:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:42:21 INFO [loop_until]: OK (rc = 0) 02:42:21 DEBUG --- stdout --- 02:42:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1858Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10150m 33% 10346Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14069m 47% 3096Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12111m 40% 10288Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4342Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12590m 42% 10412Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4358m 27% 6295Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10874m 68% 14384Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10782m 67% 14372Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4130m 25% 6691Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10766m 67% 14395Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8235m 51% 6379Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5359m 33% 7864Mi 13% 02:42:21 DEBUG --- stderr --- 02:42:21 DEBUG 02:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972581" 02:43:01 INFO [http_cmd]: http status code OK 02:43:01 DEBUG --- status code --- 02:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:01 DEBUG --- http response --- 02:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:01 DEBUG ----- output ----- 02:43:01 DEBUG { 02:43:01 DEBUG "status": "success", 02:43:01 DEBUG "data": { 02:43:01 DEBUG "resultType": "vector", 02:43:01 DEBUG "result": [ 02:43:01 DEBUG { 02:43:01 DEBUG "metric": { 02:43:01 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:01 DEBUG }, 02:43:01 DEBUG "value": [ 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG [...] 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:01 DEBUG }, 02:43:01 DEBUG "value": [ 02:43:01 DEBUG 1712972581, 02:43:01 DEBUG "1751.6333333333332" 02:43:01 DEBUG ] 02:43:01 DEBUG } 02:43:01 DEBUG ] 02:43:01 DEBUG } 02:43:01 DEBUG } 02:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972581" 02:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972581" 02:43:01 INFO [http_cmd]: http status code OK 02:43:01 INFO [http_cmd]: http status code OK 02:43:01 DEBUG --- status code --- 02:43:01 DEBUG --- status code --- 02:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:01 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:01 DEBUG --- http response --- 02:43:01 DEBUG --- http response --- 02:43:01 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:01 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:43:01 DEBUG ----- output ----- 02:43:01 DEBUG { 02:43:01 DEBUG "status": "success", 02:43:01 DEBUG "data": { 02:43:01 DEBUG "resultType": "vector", 02:43:01 DEBUG "result": [ 02:43:01 DEBUG { 02:43:01 DEBUG "metric": { 02:43:01 DEBUG "pod": "ds-idrepo-1" 02:43:01 DEBUG }, 02:43:01 DEBUG "value": [ 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG [...] 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG 02:43:01 DEBUG "pod": "ds-idrepo-0" 02:43:01 DEBUG }, 02:43:01 DEBUG "value": [ 02:43:01 DEBUG 1712972581, 02:43:01 DEBUG "0" 02:43:01 DEBUG ] 02:43:01 DEBUG } 02:43:01 DEBUG ] 02:43:01 DEBUG } 02:43:01 DEBUG } 02:43:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972581" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "instance": "10.142.0.107:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "instance": "10.142.0.77:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972581, 02:43:02 DEBUG "620709602304" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972582" 02:43:02 ERROR [http_cmd]: ERROR 02:43:02 ERROR --- status code --- 02:43:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:43:02 ERROR --- http response --- 02:43:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:43:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0.023" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "1741.5666666666666" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "1741.9333333333334" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "ds-idrepo-1" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "ds-cts-1" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0.0008814416666666667" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "instance": "10.142.0.107:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "instance": "10.142.0.77:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "51286618.03333333" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "313831" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG { 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "data": { 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG "result": [ 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG { 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:02 DEBUG "pod": "ds-idrepo-1" 02:43:02 DEBUG "pod": "ds-idrepo-1" 02:43:02 DEBUG }, 02:43:02 DEBUG }, 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG "value": [ 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG [...] 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:02 DEBUG "pod": "ds-cts-1" 02:43:02 DEBUG "pod": "ds-cts-1" 02:43:02 DEBUG }, 02:43:02 DEBUG }, 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG "value": [ 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "1741.5333333333333" 02:43:02 DEBUG "0" 02:43:02 DEBUG "29.433333333333334" 02:43:02 DEBUG ] 02:43:02 DEBUG ] 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG ] 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "ds-idrepo-1" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "ds-idrepo-0" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "ds-idrepo-1" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "ds-cts-1" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "instance": "10.142.0.107:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "instance": "10.142.0.77:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "51280836.666666664" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "402.70039981647767" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "536.9338664219703" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "ds-idrepo-1" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "ds-idrepo-0" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "instance": "10.142.0.107:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "instance": "10.142.0.77:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0.17210000000000272" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "5.240589061964648" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "6236626944" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972582" 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "instance": "10.142.0.107:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "instance": "10.142.0.77:9100" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0.04800000000000206" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972582" 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0.014171396735924246" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:02 INFO [http_cmd]: http status code OK 02:43:02 DEBUG --- status code --- 02:43:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:43:02 DEBUG --- http response --- 02:43:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:43:02 DEBUG ----- output ----- 02:43:02 DEBUG { 02:43:02 DEBUG "status": "success", 02:43:02 DEBUG "data": { 02:43:02 DEBUG "resultType": "vector", 02:43:02 DEBUG "result": [ 02:43:02 DEBUG { 02:43:02 DEBUG "metric": { 02:43:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG [...] 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG 02:43:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:43:02 DEBUG }, 02:43:02 DEBUG "value": [ 02:43:02 DEBUG 1712972582, 02:43:02 DEBUG "0" 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG ] 02:43:02 DEBUG } 02:43:02 DEBUG } 02:43:14 INFO 02:43:14 INFO [loop_until]: kubectl --namespace=xlou top pods 02:43:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:43:15 INFO [loop_until]: OK (rc = 0) 02:43:15 DEBUG --- stdout --- 02:43:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12328m 8558Mi am-6d96779b5d-qbg7l 9613m 8471Mi am-6d96779b5d-zn69c 11863m 8476Mi ds-cts-0 10756m 13506Mi ds-cts-1 11221m 13512Mi ds-cts-2 10658m 13501Mi ds-idrepo-0 8105m 5593Mi ds-idrepo-1 4302m 5522Mi ds-idrepo-2 4219m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1102Mi idm-b85f9d978-dz57p 4m 1114Mi lodemon-8dc49bd56-qr24k 8m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5370m 4122Mi 02:43:15 DEBUG --- stderr --- 02:43:15 DEBUG 02:43:21 INFO 02:43:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:43:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:43:21 INFO [loop_until]: OK (rc = 0) 02:43:21 DEBUG --- stdout --- 02:43:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10172m 34% 10353Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14089m 47% 3102Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12124m 40% 10289Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4333Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12742m 42% 10404Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4364m 27% 6299Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10857m 68% 14387Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11170m 70% 14374Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4258m 26% 6692Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10820m 68% 14454Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8218m 51% 6377Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5385m 33% 7863Mi 13% 02:43:21 DEBUG --- stderr --- 02:43:21 DEBUG 02:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972641" 02:44:01 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972641" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-0" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972641, 02:44:02 DEBUG "0" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972641, 02:44:02 DEBUG "1736.1666666666667" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "instance": "10.142.0.107:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "instance": "10.142.0.77:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "623501775872" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972642" 02:44:02 ERROR [http_cmd]: ERROR 02:44:02 ERROR --- status code --- 02:44:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:44:02 ERROR --- http response --- 02:44:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:44:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0.023" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "1746.4333333333334" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "1746.8333333333333" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG { 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG }, 02:44:02 DEBUG { 02:44:02 DEBUG "value": [ 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG 02:44:02 DEBUG "data": { 02:44:02 DEBUG 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG 02:44:02 DEBUG "result": [ 02:44:02 DEBUG [...] 02:44:02 DEBUG { 02:44:02 DEBUG 02:44:02 DEBUG "metric": { 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG 02:44:02 DEBUG }, 02:44:02 DEBUG "pod": "ds-cts-1" 02:44:02 DEBUG "value": [ 02:44:02 DEBUG }, 02:44:02 DEBUG 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG 02:44:02 DEBUG "0.001079511111111111" 02:44:02 DEBUG [...] 02:44:02 DEBUG ] 02:44:02 DEBUG 02:44:02 DEBUG } 02:44:02 DEBUG 02:44:02 DEBUG ] 02:44:02 DEBUG 02:44:02 DEBUG } 02:44:02 DEBUG "pod": "ds-cts-1" 02:44:02 DEBUG } 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "28.566666666666666" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "instance": "10.142.0.107:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "instance": "10.142.0.77:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "51516905.099999994" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "313957.5" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-cts-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "1746.8999999999999" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-cts-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-0" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "instance": "10.142.0.107:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "instance": "10.142.0.77:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "51511845.96666667" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "3152.830696994188" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-0" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "26010.853250202053" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "instance": "10.142.0.107:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "instance": "10.142.0.77:9100" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0.17216666666667302" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "5.2398379149325205" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "6236856320" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "0" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG { 02:44:02 DEBUG "status": "success", 02:44:02 DEBUG "data": { 02:44:02 DEBUG "resultType": "vector", 02:44:02 DEBUG "result": [ 02:44:02 DEBUG { 02:44:02 DEBUG "metric": { 02:44:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG [...] 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG 02:44:02 DEBUG "pod": "ds-idrepo-1" 02:44:02 DEBUG }, 02:44:02 DEBUG "value": [ 02:44:02 DEBUG 1712972642, 02:44:02 DEBUG "4.2780130143940704" 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG ] 02:44:02 DEBUG } 02:44:02 DEBUG } 02:44:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972642" 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:02 DEBUG --- http response --- 02:44:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:44:02 INFO [http_cmd]: http status code OK 02:44:02 DEBUG ----- output ----- 02:44:02 DEBUG --- status code --- 02:44:02 DEBUG { 02:44:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:44:03 DEBUG "status": "success", 02:44:03 DEBUG --- http response --- 02:44:03 DEBUG "data": { 02:44:03 DEBUG "resultType": "vector", 02:44:03 DEBUG "result": [ 02:44:03 DEBUG { 02:44:03 DEBUG "metric": { 02:44:03 DEBUG "instance": "10.142.0.107:9100" 02:44:03 DEBUG }, 02:44:03 DEBUG "value": [ 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG [...] 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG "instance": "10.142.0.77:9100" 02:44:03 DEBUG }, 02:44:03 DEBUG "value": [ 02:44:03 DEBUG 1712972642, 02:44:03 DEBUG "0.04833333333333296" 02:44:03 DEBUG ] 02:44:03 DEBUG } 02:44:03 DEBUG ] 02:44:03 DEBUG } 02:44:03 DEBUG } 02:44:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:44:03 DEBUG ----- output ----- 02:44:03 DEBUG { 02:44:03 DEBUG "status": "success", 02:44:03 DEBUG "data": { 02:44:03 DEBUG "resultType": "vector", 02:44:03 DEBUG "result": [ 02:44:03 DEBUG { 02:44:03 DEBUG "metric": { 02:44:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:44:03 DEBUG }, 02:44:03 DEBUG "value": [ 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG [...] 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG 02:44:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:44:03 DEBUG }, 02:44:03 DEBUG "value": [ 02:44:03 DEBUG 1712972642, 02:44:03 DEBUG "0" 02:44:03 DEBUG ] 02:44:03 DEBUG } 02:44:03 DEBUG ] 02:44:03 DEBUG } 02:44:03 DEBUG } 02:44:15 INFO 02:44:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:44:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:44:15 INFO [loop_until]: OK (rc = 0) 02:44:15 DEBUG --- stdout --- 02:44:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12142m 8554Mi am-6d96779b5d-qbg7l 9921m 8478Mi am-6d96779b5d-zn69c 11501m 8477Mi ds-cts-0 10474m 13510Mi ds-cts-1 10948m 13514Mi ds-cts-2 10961m 13501Mi ds-idrepo-0 8125m 5594Mi ds-idrepo-1 4320m 5522Mi ds-idrepo-2 4138m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1102Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 4m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5167m 4122Mi 02:44:15 DEBUG --- stderr --- 02:44:15 DEBUG 02:44:21 INFO 02:44:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:44:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:44:21 INFO [loop_until]: OK (rc = 0) 02:44:21 DEBUG --- stdout --- 02:44:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 4735Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1853Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10337m 34% 10351Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13686m 45% 3093Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12225m 40% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 133m 0% 4329Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12914m 43% 10407Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4279m 26% 6311Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10854m 68% 14390Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11308m 71% 14372Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4251m 26% 6691Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10880m 68% 14399Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8318m 52% 6380Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5403m 34% 7863Mi 13% 02:44:21 DEBUG --- stderr --- 02:44:21 DEBUG 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:02 DEBUG }, 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-idrepo-0" 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0" 02:45:02 DEBUG "1743.8999999999999" 02:45:02 DEBUG ] 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "instance": "10.142.0.107:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "instance": "10.142.0.77:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "626291958784" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972702" 02:45:02 ERROR [http_cmd]: ERROR 02:45:02 ERROR --- status code --- 02:45:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:45:02 ERROR --- http response --- 02:45:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:45:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "1755.3333333333333" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0.023" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "1755.7333333333333" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-cts-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0.0000666666666666667" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-cts-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "29.666666666666668" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "instance": "10.142.0.107:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "instance": "10.142.0.77:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "51451726.8" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "314421" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "1755.4666666666667" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-cts-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-cts-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-idrepo-0" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "instance": "10.142.0.107:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "instance": "10.142.0.77:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "51446211.233333334" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "ds-idrepo-1" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "ds-idrepo-0" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "7711795.624405488" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "305178438.37477916" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "instance": "10.142.0.107:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "instance": "10.142.0.77:9100" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0.18343333333333664" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG ] 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG } 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG ] 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG } 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:02 DEBUG } 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "5.285711914026565" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972702" 02:45:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972702" 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "6237122560" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:02 INFO [http_cmd]: http status code OK 02:45:02 DEBUG --- status code --- 02:45:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:02 DEBUG --- http response --- 02:45:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:45:02 DEBUG ----- output ----- 02:45:02 DEBUG { 02:45:02 DEBUG "status": "success", 02:45:02 DEBUG "data": { 02:45:02 DEBUG "resultType": "vector", 02:45:02 DEBUG "result": [ 02:45:02 DEBUG { 02:45:02 DEBUG "metric": { 02:45:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG [...] 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG 02:45:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:02 DEBUG }, 02:45:02 DEBUG "value": [ 02:45:02 DEBUG 1712972702, 02:45:02 DEBUG "0" 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG ] 02:45:02 DEBUG } 02:45:02 DEBUG } 02:45:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972703" 02:45:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972703" 02:45:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972703" 02:45:03 INFO [http_cmd]: http status code OK 02:45:03 DEBUG --- status code --- 02:45:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:03 DEBUG --- http response --- 02:45:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:45:03 DEBUG ----- output ----- 02:45:03 DEBUG { 02:45:03 DEBUG "status": "success", 02:45:03 DEBUG "data": { 02:45:03 DEBUG "resultType": "vector", 02:45:03 DEBUG "result": [ 02:45:03 DEBUG { 02:45:03 DEBUG "metric": { 02:45:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:45:03 DEBUG }, 02:45:03 DEBUG "value": [ 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG [...] 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:03 DEBUG }, 02:45:03 DEBUG "value": [ 02:45:03 DEBUG 1712972703, 02:45:03 DEBUG "0" 02:45:03 DEBUG ] 02:45:03 DEBUG } 02:45:03 DEBUG ] 02:45:03 DEBUG } 02:45:03 DEBUG } 02:45:03 INFO [http_cmd]: http status code OK 02:45:03 DEBUG --- status code --- 02:45:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:03 DEBUG --- http response --- 02:45:03 INFO [http_cmd]: http status code OK 02:45:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:45:03 DEBUG --- status code --- 02:45:03 DEBUG ----- output ----- 02:45:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:45:03 DEBUG { 02:45:03 DEBUG --- http response --- 02:45:03 DEBUG "status": "success", 02:45:03 DEBUG "data": { 02:45:03 DEBUG "resultType": "vector", 02:45:03 DEBUG "result": [ 02:45:03 DEBUG { 02:45:03 DEBUG "metric": { 02:45:03 DEBUG "instance": "10.142.0.107:9100" 02:45:03 DEBUG }, 02:45:03 DEBUG "value": [ 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG [...] 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG "instance": "10.142.0.77:9100" 02:45:03 DEBUG }, 02:45:03 DEBUG "value": [ 02:45:03 DEBUG 1712972703, 02:45:03 DEBUG "0.05633333333333397" 02:45:03 DEBUG ] 02:45:03 DEBUG } 02:45:03 DEBUG ] 02:45:03 DEBUG } 02:45:03 DEBUG } 02:45:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:45:03 DEBUG ----- output ----- 02:45:03 DEBUG { 02:45:03 DEBUG "status": "success", 02:45:03 DEBUG "data": { 02:45:03 DEBUG "resultType": "vector", 02:45:03 DEBUG "result": [ 02:45:03 DEBUG { 02:45:03 DEBUG "metric": { 02:45:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:45:03 DEBUG }, 02:45:03 DEBUG "value": [ 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG [...] 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG 02:45:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:45:03 DEBUG }, 02:45:03 DEBUG "value": [ 02:45:03 DEBUG 1712972703, 02:45:03 DEBUG "12.131799123522548" 02:45:03 DEBUG ] 02:45:03 DEBUG } 02:45:03 DEBUG ] 02:45:03 DEBUG } 02:45:03 DEBUG } 02:45:15 INFO 02:45:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:45:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:45:15 INFO [loop_until]: OK (rc = 0) 02:45:15 DEBUG --- stdout --- 02:45:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12252m 8565Mi am-6d96779b5d-qbg7l 9769m 8473Mi am-6d96779b5d-zn69c 11847m 8482Mi ds-cts-0 10868m 13597Mi ds-cts-1 11204m 13662Mi ds-cts-2 11029m 13501Mi ds-idrepo-0 7941m 5594Mi ds-idrepo-1 4325m 5522Mi ds-idrepo-2 4035m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1103Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5284m 4123Mi 02:45:15 DEBUG --- stderr --- 02:45:15 DEBUG 02:45:21 INFO 02:45:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:45:21 INFO [loop_until]: OK (rc = 0) 02:45:21 DEBUG --- stdout --- 02:45:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 4746Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10252m 34% 10352Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14096m 47% 3089Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12063m 40% 10282Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 4353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12766m 42% 10410Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4411m 27% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10540m 66% 14396Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11310m 71% 14377Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4242m 26% 6693Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10981m 69% 14405Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8254m 51% 6386Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5380m 33% 7864Mi 13% 02:45:21 DEBUG --- stderr --- 02:45:21 DEBUG 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "ds-idrepo-0" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "1732.0666666666666" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "instance": "10.142.0.107:9100" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "instance": "10.142.0.77:9100" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "629079911424" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 ERROR [http_cmd]: ERROR 02:46:02 ERROR --- status code --- 02:46:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:46:02 ERROR --- http response --- 02:46:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:46:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0.023" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "1754.1" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "1750.8333333333333" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "314617" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG --- http response --- 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG { 02:46:02 DEBUG { 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "data": { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "result": [ 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG { 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG "metric": { 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "instance": "10.142.0.107:9100" 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG }, 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG "value": [ 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "ds-cts-1" 02:46:02 DEBUG [...] 02:46:02 DEBUG "pod": "ds-cts-1" 02:46:02 DEBUG }, 02:46:02 DEBUG 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG "instance": "10.142.0.77:9100" 02:46:02 DEBUG "29.333333333333332" 02:46:02 DEBUG ] 02:46:02 DEBUG }, 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG "value": [ 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG "51529719.733333334" 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:02 DEBUG }, 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:46:02 DEBUG "value": [ 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG 02:46:02 DEBUG { 02:46:02 DEBUG 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG 02:46:02 DEBUG "data": { 02:46:02 DEBUG [...] 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG 02:46:02 DEBUG "result": [ 02:46:02 DEBUG 02:46:02 DEBUG { 02:46:02 DEBUG 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG }, 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG 02:46:02 DEBUG "1750.5333333333333" 02:46:02 DEBUG 02:46:02 DEBUG ] 02:46:02 DEBUG 02:46:02 DEBUG } 02:46:02 DEBUG [...] 02:46:02 DEBUG ] 02:46:02 DEBUG 02:46:02 DEBUG } 02:46:02 DEBUG 02:46:02 DEBUG } 02:46:02 DEBUG 02:46:02 DEBUG "pod": "ds-cts-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "ds-cts-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "ds-idrepo-0" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG { 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "instance": "10.142.0.107:9100" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "instance": "10.142.0.77:9100" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "51526847.599999994" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "ds-idrepo-1" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "ds-idrepo-0" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "instance": "10.142.0.107:9100" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "instance": "10.142.0.77:9100" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0.17613333333332973" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "5.261862943564096" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972762" 02:46:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972762" 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "6237622272" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:02 INFO [http_cmd]: http status code OK 02:46:02 DEBUG --- status code --- 02:46:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:02 DEBUG --- http response --- 02:46:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:02 DEBUG ----- output ----- 02:46:02 DEBUG { 02:46:02 DEBUG "status": "success", 02:46:02 DEBUG "data": { 02:46:02 DEBUG "resultType": "vector", 02:46:02 DEBUG "result": [ 02:46:02 DEBUG { 02:46:02 DEBUG "metric": { 02:46:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG [...] 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG 02:46:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:46:02 DEBUG }, 02:46:02 DEBUG "value": [ 02:46:02 DEBUG 1712972762, 02:46:02 DEBUG "0" 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG ] 02:46:02 DEBUG } 02:46:02 DEBUG } 02:46:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972763" 02:46:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972763" 02:46:03 INFO [http_cmd]: http status code OK 02:46:03 DEBUG --- status code --- 02:46:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:03 DEBUG --- http response --- 02:46:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:03 DEBUG ----- output ----- 02:46:03 DEBUG { 02:46:03 DEBUG "status": "success", 02:46:03 DEBUG "data": { 02:46:03 DEBUG "resultType": "vector", 02:46:03 DEBUG "result": [ 02:46:03 DEBUG { 02:46:03 DEBUG "metric": { 02:46:03 DEBUG "instance": "10.142.0.107:9100" 02:46:03 DEBUG }, 02:46:03 DEBUG "value": [ 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG [...] 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG "instance": "10.142.0.77:9100" 02:46:03 DEBUG }, 02:46:03 DEBUG "value": [ 02:46:03 DEBUG 1712972763, 02:46:03 DEBUG "0.05166666666666443" 02:46:03 DEBUG ] 02:46:03 DEBUG } 02:46:03 DEBUG ] 02:46:03 DEBUG } 02:46:03 DEBUG } 02:46:03 INFO [http_cmd]: http status code OK 02:46:03 DEBUG --- status code --- 02:46:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:03 DEBUG --- http response --- 02:46:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:46:03 DEBUG ----- output ----- 02:46:03 DEBUG { 02:46:03 DEBUG "status": "success", 02:46:03 DEBUG "data": { 02:46:03 DEBUG "resultType": "vector", 02:46:03 DEBUG "result": [ 02:46:03 DEBUG { 02:46:03 DEBUG "metric": { 02:46:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:46:03 DEBUG }, 02:46:03 DEBUG "value": [ 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG [...] 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:46:03 DEBUG }, 02:46:03 DEBUG "value": [ 02:46:03 DEBUG 1712972763, 02:46:03 DEBUG "0" 02:46:03 DEBUG ] 02:46:03 DEBUG } 02:46:03 DEBUG ] 02:46:03 DEBUG } 02:46:03 DEBUG } 02:46:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972763" 02:46:03 INFO [http_cmd]: http status code OK 02:46:03 DEBUG --- status code --- 02:46:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:46:03 DEBUG --- http response --- 02:46:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:46:03 DEBUG ----- output ----- 02:46:03 DEBUG { 02:46:03 DEBUG "status": "success", 02:46:03 DEBUG "data": { 02:46:03 DEBUG "resultType": "vector", 02:46:03 DEBUG "result": [ 02:46:03 DEBUG { 02:46:03 DEBUG "metric": { 02:46:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:46:03 DEBUG }, 02:46:03 DEBUG "value": [ 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG [...] 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG 02:46:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:46:03 DEBUG }, 02:46:03 DEBUG "value": [ 02:46:03 DEBUG 1712972763, 02:46:03 DEBUG "10.430189527227755" 02:46:03 DEBUG ] 02:46:03 DEBUG } 02:46:03 DEBUG ] 02:46:03 DEBUG } 02:46:03 DEBUG } 02:46:15 INFO 02:46:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:46:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:46:15 INFO [loop_until]: OK (rc = 0) 02:46:15 DEBUG --- stdout --- 02:46:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12178m 8567Mi am-6d96779b5d-qbg7l 9695m 8474Mi am-6d96779b5d-zn69c 11980m 8479Mi ds-cts-0 10655m 13687Mi ds-cts-1 11226m 13644Mi ds-cts-2 10919m 13500Mi ds-idrepo-0 8186m 5594Mi ds-idrepo-1 4288m 5522Mi ds-idrepo-2 4220m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 6m 1103Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5190m 4123Mi 02:46:15 DEBUG --- stderr --- 02:46:15 DEBUG 02:46:21 INFO 02:46:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:46:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:46:21 INFO [loop_until]: OK (rc = 0) 02:46:21 DEBUG --- stdout --- 02:46:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4740Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 1860Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10311m 34% 10358Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13982m 46% 3089Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12001m 40% 10283Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4346Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12880m 43% 10410Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4409m 27% 6300Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10895m 68% 14398Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10785m 67% 14381Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4152m 26% 6695Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10934m 68% 14412Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 7968m 50% 6382Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5384m 33% 7875Mi 13% 02:46:21 DEBUG --- stderr --- 02:46:21 DEBUG 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-idrepo-0" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "0" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "1747.3333333333333" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 ERROR [http_cmd]: ERROR 02:47:02 ERROR --- status code --- 02:47:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:47:02 ERROR --- http response --- 02:47:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "instance": "10.142.0.107:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "instance": "10.142.0.77:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "631871699968" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "0.023" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "1737.1666666666667" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "1736.7" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "314456" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-cts-1" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "0.0012500777777777778" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-cts-1" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "29.333333333333332" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "instance": "10.142.0.107:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "instance": "10.142.0.77:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "51193583.43333334" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG }, 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:02 DEBUG "pod": "ds-cts-1" 02:47:02 DEBUG }, 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "1736.2333333333333" 02:47:02 DEBUG "0" 02:47:02 DEBUG ] 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG }, 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-idrepo-0" 02:47:02 DEBUG "pod": "ds-cts-1" 02:47:02 DEBUG }, 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "0" 02:47:02 DEBUG "0" 02:47:02 DEBUG ] 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972822" 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG { 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "instance": "10.142.0.107:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "instance": "10.142.0.77:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "51192009.46666667" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-cts-2" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "47084141.293808" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "ds-idrepo-1" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-idrepo-0" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "0" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "pod": "ds-cts-2" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "238020.7785011774" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "instance": "10.142.0.107:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "instance": "10.142.0.77:9100" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "0.1931999999999898" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972822" 02:47:02 INFO [http_cmd]: http status code OK 02:47:02 DEBUG --- status code --- 02:47:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:02 DEBUG --- http response --- 02:47:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:02 DEBUG ----- output ----- 02:47:02 DEBUG { 02:47:02 DEBUG "status": "success", 02:47:02 DEBUG "data": { 02:47:02 DEBUG "resultType": "vector", 02:47:02 DEBUG "result": [ 02:47:02 DEBUG { 02:47:02 DEBUG "metric": { 02:47:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG [...] 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG 02:47:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:47:02 DEBUG }, 02:47:02 DEBUG "value": [ 02:47:02 DEBUG 1712972822, 02:47:02 DEBUG "5.313947752321525" 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG ] 02:47:02 DEBUG } 02:47:02 DEBUG } 02:47:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972823" 02:47:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972823" 02:47:03 INFO [http_cmd]: http status code OK 02:47:03 DEBUG --- status code --- 02:47:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:03 DEBUG --- http response --- 02:47:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:47:03 DEBUG ----- output ----- 02:47:03 DEBUG { 02:47:03 DEBUG "status": "success", 02:47:03 DEBUG "data": { 02:47:03 DEBUG "resultType": "vector", 02:47:03 DEBUG "result": [ 02:47:03 DEBUG { 02:47:03 DEBUG "metric": { 02:47:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG [...] 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG "pod": "ds-cts-2" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 1712972823, 02:47:03 DEBUG "34646066.21415708" 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG } 02:47:03 INFO [http_cmd]: http status code OK 02:47:03 DEBUG --- status code --- 02:47:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:03 DEBUG --- http response --- 02:47:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:03 DEBUG ----- output ----- 02:47:03 DEBUG { 02:47:03 DEBUG "status": "success", 02:47:03 DEBUG "data": { 02:47:03 DEBUG "resultType": "vector", 02:47:03 DEBUG "result": [ 02:47:03 DEBUG { 02:47:03 DEBUG "metric": { 02:47:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG [...] 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 1712972823, 02:47:03 DEBUG "6237130752" 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG } 02:47:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972823" 02:47:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972823" 02:47:03 INFO [http_cmd]: http status code OK 02:47:03 DEBUG --- status code --- 02:47:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:03 DEBUG --- http response --- 02:47:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:47:03 DEBUG ----- output ----- 02:47:03 DEBUG { 02:47:03 DEBUG "status": "success", 02:47:03 DEBUG "data": { 02:47:03 DEBUG "resultType": "vector", 02:47:03 DEBUG "result": [ 02:47:03 DEBUG { 02:47:03 DEBUG "metric": { 02:47:03 DEBUG "instance": "10.142.0.107:9100" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG [...] 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG "instance": "10.142.0.77:9100" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 1712972823, 02:47:03 DEBUG "0.06233333333333395" 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG } 02:47:03 INFO [http_cmd]: http status code OK 02:47:03 DEBUG --- status code --- 02:47:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:03 DEBUG --- http response --- 02:47:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:47:03 DEBUG ----- output ----- 02:47:03 DEBUG { 02:47:03 DEBUG "status": "success", 02:47:03 DEBUG "data": { 02:47:03 DEBUG "resultType": "vector", 02:47:03 DEBUG "result": [ 02:47:03 DEBUG { 02:47:03 DEBUG "metric": { 02:47:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG [...] 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 1712972823, 02:47:03 DEBUG "0" 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG } 02:47:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972823" 02:47:03 INFO [http_cmd]: http status code OK 02:47:03 DEBUG --- status code --- 02:47:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:47:03 DEBUG --- http response --- 02:47:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:47:03 DEBUG ----- output ----- 02:47:03 DEBUG { 02:47:03 DEBUG "status": "success", 02:47:03 DEBUG "data": { 02:47:03 DEBUG "resultType": "vector", 02:47:03 DEBUG "result": [ 02:47:03 DEBUG { 02:47:03 DEBUG "metric": { 02:47:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG [...] 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG 02:47:03 DEBUG "pod": "ds-cts-2" 02:47:03 DEBUG }, 02:47:03 DEBUG "value": [ 02:47:03 DEBUG 1712972823, 02:47:03 DEBUG "10.849881736147621" 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG ] 02:47:03 DEBUG } 02:47:03 DEBUG } 02:47:15 INFO 02:47:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:47:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:47:15 INFO [loop_until]: OK (rc = 0) 02:47:15 DEBUG --- stdout --- 02:47:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11926m 8568Mi am-6d96779b5d-qbg7l 10095m 8477Mi am-6d96779b5d-zn69c 11562m 8481Mi ds-cts-0 10759m 13642Mi ds-cts-1 11023m 13581Mi ds-cts-2 10776m 13500Mi ds-idrepo-0 8163m 5594Mi ds-idrepo-1 4310m 5522Mi ds-idrepo-2 4212m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1103Mi idm-b85f9d978-dz57p 3m 1115Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5241m 4122Mi 02:47:15 DEBUG --- stderr --- 02:47:15 DEBUG 02:47:21 INFO 02:47:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:47:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:47:21 INFO [loop_until]: OK (rc = 0) 02:47:21 DEBUG --- stdout --- 02:47:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4728Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10257m 34% 10361Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14159m 47% 3106Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12125m 40% 10286Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 4355Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12834m 42% 10412Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4368m 27% 6302Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10847m 68% 14410Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11405m 71% 14382Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4318m 27% 6694Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10946m 68% 14412Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8256m 51% 6382Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5399m 33% 7876Mi 13% 02:47:21 DEBUG --- stderr --- 02:47:21 DEBUG 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-idrepo-0" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "1783.5666666666666" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972882" 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 ERROR [http_cmd]: ERROR 02:48:02 ERROR --- status code --- 02:48:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:48:02 ERROR --- http response --- 02:48:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:48:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0.023" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "instance": "10.142.0.107:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "instance": "10.142.0.77:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "634658675712" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "1739.1333333333334" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "1750.9333333333334" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "314821" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-cts-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0.00003333333333333336" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-cts-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "29.966666666666665" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "instance": "10.142.0.107:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "instance": "10.142.0.77:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "51487932.3" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-cts-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "1750.7666666666667" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-cts-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-idrepo-0" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972882" 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "instance": "10.142.0.107:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "instance": "10.142.0.77:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "51484148.833333336" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "ds-idrepo-1" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "ds-idrepo-0" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "pod": "idm-b85f9d978-b559t" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "32283189.105261557" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "instance": "10.142.0.107:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "instance": "10.142.0.77:9100" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "0.19013333333333882" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972882" 02:48:02 INFO [http_cmd]: http status code OK 02:48:02 DEBUG --- status code --- 02:48:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:02 DEBUG --- http response --- 02:48:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:02 DEBUG ----- output ----- 02:48:02 DEBUG { 02:48:02 DEBUG "status": "success", 02:48:02 DEBUG "data": { 02:48:02 DEBUG "resultType": "vector", 02:48:02 DEBUG "result": [ 02:48:02 DEBUG { 02:48:02 DEBUG "metric": { 02:48:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG [...] 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG 02:48:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:48:02 DEBUG }, 02:48:02 DEBUG "value": [ 02:48:02 DEBUG 1712972882, 02:48:02 DEBUG "5.275258646127013" 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG ] 02:48:02 DEBUG } 02:48:02 DEBUG } 02:48:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972883" 02:48:03 INFO [http_cmd]: http status code OK 02:48:03 DEBUG --- status code --- 02:48:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:03 DEBUG --- http response --- 02:48:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:48:03 DEBUG ----- output ----- 02:48:03 DEBUG { 02:48:03 DEBUG "status": "success", 02:48:03 DEBUG "data": { 02:48:03 DEBUG "resultType": "vector", 02:48:03 DEBUG "result": [ 02:48:03 DEBUG { 02:48:03 DEBUG "metric": { 02:48:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG [...] 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG "pod": "idm-b85f9d978-b559t" 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 1712972883, 02:48:03 DEBUG "0" 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG } 02:48:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972883" 02:48:03 INFO [http_cmd]: http status code OK 02:48:03 DEBUG --- status code --- 02:48:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:03 DEBUG --- http response --- 02:48:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:03 DEBUG ----- output ----- 02:48:03 DEBUG { 02:48:03 DEBUG "status": "success", 02:48:03 DEBUG "data": { 02:48:03 DEBUG "resultType": "vector", 02:48:03 DEBUG "result": [ 02:48:03 DEBUG { 02:48:03 DEBUG "metric": { 02:48:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG [...] 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 1712972883, 02:48:03 DEBUG "6238629888" 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG } 02:48:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972883" 02:48:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972883" 02:48:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972883" 02:48:03 INFO [http_cmd]: http status code OK 02:48:03 DEBUG --- status code --- 02:48:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:03 DEBUG --- http response --- 02:48:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:48:03 DEBUG ----- output ----- 02:48:03 DEBUG { 02:48:03 DEBUG "status": "success", 02:48:03 DEBUG "data": { 02:48:03 DEBUG "resultType": "vector", 02:48:03 DEBUG "result": [ 02:48:03 DEBUG { 02:48:03 DEBUG "metric": { 02:48:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG [...] 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG 02:48:03 DEBUG "pod": "idm-b85f9d978-b559t" 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 1712972883, 02:48:03 DEBUG "0.0018885721239391912" 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG } 02:48:03 INFO [http_cmd]: http status code OK 02:48:03 INFO [http_cmd]: http status code OK 02:48:03 DEBUG --- status code --- 02:48:03 DEBUG --- status code --- 02:48:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:48:03 DEBUG --- http response --- 02:48:03 DEBUG --- http response --- 02:48:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:48:03 DEBUG ----- output ----- 02:48:03 DEBUG { 02:48:03 DEBUG "status": "success", 02:48:03 DEBUG "data": { 02:48:03 DEBUG "resultType": "vector", 02:48:03 DEBUG "result": [ 02:48:03 DEBUG { 02:48:03 DEBUG "metric": { 02:48:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:48:03 DEBUG "instance": "10.142.0.107:9100" 02:48:03 DEBUG ----- output ----- 02:48:03 DEBUG }, 02:48:03 DEBUG { 02:48:03 DEBUG "value": [ 02:48:03 DEBUG "status": "success", 02:48:03 DEBUG 02:48:03 DEBUG "data": { 02:48:03 DEBUG 02:48:03 DEBUG "resultType": "vector", 02:48:03 DEBUG 02:48:03 DEBUG "result": [ 02:48:03 DEBUG [...] 02:48:03 DEBUG { 02:48:03 DEBUG 02:48:03 DEBUG "metric": { 02:48:03 DEBUG 02:48:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:48:03 DEBUG 02:48:03 DEBUG }, 02:48:03 DEBUG "instance": "10.142.0.77:9100" 02:48:03 DEBUG "value": [ 02:48:03 DEBUG }, 02:48:03 DEBUG 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 02:48:03 DEBUG 1712972883, 02:48:03 DEBUG 02:48:03 DEBUG "0.06433333333333426" 02:48:03 DEBUG [...] 02:48:03 DEBUG ] 02:48:03 DEBUG 02:48:03 DEBUG } 02:48:03 DEBUG 02:48:03 DEBUG ] 02:48:03 DEBUG 02:48:03 DEBUG } 02:48:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:48:03 DEBUG } 02:48:03 DEBUG }, 02:48:03 DEBUG "value": [ 02:48:03 DEBUG 1712972883, 02:48:03 DEBUG "0" 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG ] 02:48:03 DEBUG } 02:48:03 DEBUG } 02:48:15 INFO 02:48:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:48:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:48:15 INFO [loop_until]: OK (rc = 0) 02:48:15 DEBUG --- stdout --- 02:48:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12382m 8567Mi am-6d96779b5d-qbg7l 9811m 8472Mi am-6d96779b5d-zn69c 11504m 8480Mi ds-cts-0 10901m 13637Mi ds-cts-1 11492m 13624Mi ds-cts-2 11159m 13508Mi ds-idrepo-0 8036m 5594Mi ds-idrepo-1 4268m 5522Mi ds-idrepo-2 4230m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1103Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 8m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5164m 4123Mi 02:48:15 DEBUG --- stderr --- 02:48:15 DEBUG 02:48:21 INFO 02:48:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:48:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:48:21 INFO [loop_until]: OK (rc = 0) 02:48:21 DEBUG --- stdout --- 02:48:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4747Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1865Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10259m 34% 10357Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13631m 45% 3111Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12137m 40% 10282Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 4340Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12597m 42% 10411Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4343m 27% 6301Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10680m 67% 14425Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11004m 69% 14364Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4232m 26% 6697Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10551m 66% 14398Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8260m 51% 6382Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5365m 33% 7861Mi 13% 02:48:21 DEBUG --- stderr --- 02:48:21 DEBUG 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-idrepo-0" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "0" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "data": { 02:49:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG }, 02:49:02 DEBUG "result": [ 02:49:02 DEBUG "value": [ 02:49:02 DEBUG { 02:49:02 DEBUG 02:49:02 DEBUG "metric": { 02:49:02 DEBUG 02:49:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:02 DEBUG 02:49:02 DEBUG }, 02:49:02 DEBUG [...] 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:02 DEBUG [...] 02:49:02 DEBUG }, 02:49:02 DEBUG 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG 02:49:02 DEBUG "1755.5" 02:49:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:02 DEBUG ] 02:49:02 DEBUG }, 02:49:02 DEBUG } 02:49:02 DEBUG "value": [ 02:49:02 DEBUG ] 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG } 02:49:02 DEBUG "0.023" 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712972942" 02:49:02 ERROR [http_cmd]: ERROR 02:49:02 ERROR --- status code --- 02:49:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:49:02 ERROR --- http response --- 02:49:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:49:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "instance": "10.142.0.107:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "instance": "10.142.0.77:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "637427905536" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "1738" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "314348" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "1737.6666666666667" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-cts-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "0.0014590222222222223" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "instance": "10.142.0.107:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "instance": "10.142.0.77:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "51534395.43333334" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-cts-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "27.5" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG { 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:49:02 DEBUG "data": { 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG }, 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "value": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG 02:49:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:02 DEBUG }, 02:49:02 DEBUG 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG [...] 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG "pod": "ds-cts-1" 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG }, 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-cts-1" 02:49:02 DEBUG "0" 02:49:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:02 DEBUG }, 02:49:02 DEBUG ] 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG } 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG ] 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "0" 02:49:02 DEBUG } 02:49:02 DEBUG "1737.6" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-cts-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "33188633.26832822" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-idrepo-0" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "0" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972942" 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "instance": "10.142.0.107:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "instance": "10.142.0.77:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "51530675.43333334" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-cts-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "3750077.412560487" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "pod": "ds-idrepo-1" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "pod": "ds-idrepo-0" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "0" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "instance": "10.142.0.107:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "instance": "10.142.0.77:9100" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "0.15806666666665686" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712972942" 02:49:02 INFO [http_cmd]: http status code OK 02:49:02 DEBUG --- status code --- 02:49:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:02 DEBUG --- http response --- 02:49:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:02 DEBUG ----- output ----- 02:49:02 DEBUG { 02:49:02 DEBUG "status": "success", 02:49:02 DEBUG "data": { 02:49:02 DEBUG "resultType": "vector", 02:49:02 DEBUG "result": [ 02:49:02 DEBUG { 02:49:02 DEBUG "metric": { 02:49:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG [...] 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG 02:49:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:49:02 DEBUG }, 02:49:02 DEBUG "value": [ 02:49:02 DEBUG 1712972942, 02:49:02 DEBUG "5.253834983090963" 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG ] 02:49:02 DEBUG } 02:49:02 DEBUG } 02:49:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972943" 02:49:03 INFO [http_cmd]: http status code OK 02:49:03 DEBUG --- status code --- 02:49:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:03 DEBUG --- http response --- 02:49:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:49:03 DEBUG ----- output ----- 02:49:03 DEBUG { 02:49:03 DEBUG "status": "success", 02:49:03 DEBUG "data": { 02:49:03 DEBUG "resultType": "vector", 02:49:03 DEBUG "result": [ 02:49:03 DEBUG { 02:49:03 DEBUG "metric": { 02:49:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG [...] 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG "pod": "ds-cts-1" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 1712972943, 02:49:03 DEBUG "25257182.242294956" 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG } 02:49:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712972943" 02:49:03 INFO [http_cmd]: http status code OK 02:49:03 DEBUG --- status code --- 02:49:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:03 DEBUG --- http response --- 02:49:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:03 DEBUG ----- output ----- 02:49:03 DEBUG { 02:49:03 DEBUG "status": "success", 02:49:03 DEBUG "data": { 02:49:03 DEBUG "resultType": "vector", 02:49:03 DEBUG "result": [ 02:49:03 DEBUG { 02:49:03 DEBUG "metric": { 02:49:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG [...] 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 1712972943, 02:49:03 DEBUG "6239072256" 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG } 02:49:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972943" 02:49:03 INFO [http_cmd]: http status code OK 02:49:03 DEBUG --- status code --- 02:49:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:03 DEBUG --- http response --- 02:49:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:49:03 DEBUG ----- output ----- 02:49:03 DEBUG { 02:49:03 DEBUG "status": "success", 02:49:03 DEBUG "data": { 02:49:03 DEBUG "resultType": "vector", 02:49:03 DEBUG "result": [ 02:49:03 DEBUG { 02:49:03 DEBUG "metric": { 02:49:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG [...] 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG "pod": "ds-cts-1" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 1712972943, 02:49:03 DEBUG "8.201856008137382" 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG } 02:49:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712972943" 02:49:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712972943" 02:49:03 INFO [http_cmd]: http status code OK 02:49:03 DEBUG --- status code --- 02:49:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:03 DEBUG --- http response --- 02:49:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:49:03 DEBUG ----- output ----- 02:49:03 DEBUG { 02:49:03 DEBUG "status": "success", 02:49:03 DEBUG "data": { 02:49:03 DEBUG "resultType": "vector", 02:49:03 DEBUG "result": [ 02:49:03 DEBUG { 02:49:03 DEBUG "metric": { 02:49:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG [...] 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 1712972943, 02:49:03 DEBUG "0" 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG } 02:49:03 INFO [http_cmd]: http status code OK 02:49:03 DEBUG --- status code --- 02:49:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:49:03 DEBUG --- http response --- 02:49:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:49:03 DEBUG ----- output ----- 02:49:03 DEBUG { 02:49:03 DEBUG "status": "success", 02:49:03 DEBUG "data": { 02:49:03 DEBUG "resultType": "vector", 02:49:03 DEBUG "result": [ 02:49:03 DEBUG { 02:49:03 DEBUG "metric": { 02:49:03 DEBUG "instance": "10.142.0.107:9100" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG [...] 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG 02:49:03 DEBUG "instance": "10.142.0.77:9100" 02:49:03 DEBUG }, 02:49:03 DEBUG "value": [ 02:49:03 DEBUG 1712972943, 02:49:03 DEBUG "0.045666666666668715" 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG ] 02:49:03 DEBUG } 02:49:03 DEBUG } 02:49:15 INFO 02:49:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:49:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:49:15 INFO [loop_until]: OK (rc = 0) 02:49:15 DEBUG --- stdout --- 02:49:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11958m 8570Mi am-6d96779b5d-qbg7l 9841m 8472Mi am-6d96779b5d-zn69c 11722m 8487Mi ds-cts-0 10722m 13625Mi ds-cts-1 11267m 13512Mi ds-cts-2 10619m 13518Mi ds-idrepo-0 8266m 5594Mi ds-idrepo-1 4247m 5522Mi ds-idrepo-2 4149m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1103Mi idm-b85f9d978-dz57p 4m 1114Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5148m 4123Mi 02:49:15 DEBUG --- stderr --- 02:49:15 DEBUG 02:49:21 INFO 02:49:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:49:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:49:21 INFO [loop_until]: OK (rc = 0) 02:49:21 DEBUG --- stdout --- 02:49:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4732Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1879Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10197m 34% 10361Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14112m 47% 3105Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11749m 39% 10290Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 4327Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12935m 43% 10417Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4263m 26% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10458m 65% 14420Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11591m 72% 14502Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4236m 26% 6694Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10472m 65% 14457Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8283m 52% 6382Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5375m 33% 7863Mi 13% 02:49:21 DEBUG --- stderr --- 02:49:21 DEBUG 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-idrepo-0" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "1732.8666666666666" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0.023" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973002" 02:50:02 ERROR [http_cmd]: ERROR 02:50:02 ERROR --- status code --- 02:50:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:50:02 ERROR --- http response --- 02:50:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:50:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "instance": "10.142.0.107:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "instance": "10.142.0.77:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "640129854464" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "1734.6666666666667" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-cts-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0.00017115833333333335" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "314188.5" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "1732.8" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973002" 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "instance": "10.142.0.107:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "instance": "10.142.0.77:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "50885184.03333333" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-cts-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "29.733333333333334" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-cts-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:50:02 DEBUG "metric": { 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:02 DEBUG { 02:50:02 DEBUG }, 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "value": [ 02:50:02 DEBUG "data": { 02:50:02 DEBUG 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG 02:50:02 DEBUG "result": [ 02:50:02 DEBUG 02:50:02 DEBUG { 02:50:02 DEBUG [...] 02:50:02 DEBUG "metric": { 02:50:02 DEBUG 02:50:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:50:02 DEBUG 02:50:02 DEBUG }, 02:50:02 DEBUG 02:50:02 DEBUG "value": [ 02:50:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:02 DEBUG 02:50:02 DEBUG }, 02:50:02 DEBUG 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG [...] 02:50:02 DEBUG "1733.4666666666667" 02:50:02 DEBUG 02:50:02 DEBUG ] 02:50:02 DEBUG 02:50:02 DEBUG } 02:50:02 DEBUG 02:50:02 DEBUG ] 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG } 02:50:02 DEBUG }, 02:50:02 DEBUG } 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "2956.8669915177766" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-cts-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-idrepo-0" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973002" 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973002" 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "instance": "10.142.0.107:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "instance": "10.142.0.77:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "50881321.766666666" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "1795.2406734215072" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "pod": "ds-idrepo-1" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "pod": "ds-idrepo-0" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "instance": "10.142.0.107:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "instance": "10.142.0.77:9100" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "0.1613999999999995" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973002" 02:50:02 INFO [http_cmd]: http status code OK 02:50:02 DEBUG --- status code --- 02:50:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:02 DEBUG --- http response --- 02:50:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:02 DEBUG ----- output ----- 02:50:02 DEBUG { 02:50:02 DEBUG "status": "success", 02:50:02 DEBUG "data": { 02:50:02 DEBUG "resultType": "vector", 02:50:02 DEBUG "result": [ 02:50:02 DEBUG { 02:50:02 DEBUG "metric": { 02:50:02 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG [...] 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG 02:50:02 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:50:02 DEBUG }, 02:50:02 DEBUG "value": [ 02:50:02 DEBUG 1712973002, 02:50:02 DEBUG "5.263888893999748" 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG ] 02:50:02 DEBUG } 02:50:02 DEBUG } 02:50:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973003" 02:50:03 INFO [http_cmd]: http status code OK 02:50:03 DEBUG --- status code --- 02:50:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:03 DEBUG --- http response --- 02:50:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:50:03 DEBUG ----- output ----- 02:50:03 DEBUG { 02:50:03 DEBUG "status": "success", 02:50:03 DEBUG "data": { 02:50:03 DEBUG "resultType": "vector", 02:50:03 DEBUG "result": [ 02:50:03 DEBUG { 02:50:03 DEBUG "metric": { 02:50:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG [...] 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG "pod": "ds-idrepo-1" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 1712973003, 02:50:03 DEBUG "0" 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG } 02:50:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973003" 02:50:03 INFO [http_cmd]: http status code OK 02:50:03 DEBUG --- status code --- 02:50:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:03 DEBUG --- http response --- 02:50:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:03 DEBUG ----- output ----- 02:50:03 DEBUG { 02:50:03 DEBUG "status": "success", 02:50:03 DEBUG "data": { 02:50:03 DEBUG "resultType": "vector", 02:50:03 DEBUG "result": [ 02:50:03 DEBUG { 02:50:03 DEBUG "metric": { 02:50:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG [...] 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 1712973003, 02:50:03 DEBUG "6237736960" 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG } 02:50:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973003" 02:50:03 INFO [http_cmd]: http status code OK 02:50:03 DEBUG --- status code --- 02:50:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:03 DEBUG --- http response --- 02:50:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:50:03 DEBUG ----- output ----- 02:50:03 DEBUG { 02:50:03 DEBUG "status": "success", 02:50:03 DEBUG "data": { 02:50:03 DEBUG "resultType": "vector", 02:50:03 DEBUG "result": [ 02:50:03 DEBUG { 02:50:03 DEBUG "metric": { 02:50:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG [...] 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG "pod": "ds-idrepo-1" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 1712973003, 02:50:03 DEBUG "4.265048347745399" 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG } 02:50:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973003" 02:50:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973003" 02:50:03 INFO [http_cmd]: http status code OK 02:50:03 DEBUG --- status code --- 02:50:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:03 DEBUG --- http response --- 02:50:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:50:03 DEBUG ----- output ----- 02:50:03 DEBUG { 02:50:03 DEBUG "status": "success", 02:50:03 DEBUG "data": { 02:50:03 DEBUG "resultType": "vector", 02:50:03 DEBUG "result": [ 02:50:03 DEBUG { 02:50:03 DEBUG "metric": { 02:50:03 DEBUG "instance": "10.142.0.107:9100" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG [...] 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG "instance": "10.142.0.77:9100" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 1712973003, 02:50:03 DEBUG "0.048666666666665026" 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG } 02:50:03 INFO [http_cmd]: http status code OK 02:50:03 DEBUG --- status code --- 02:50:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:50:03 DEBUG --- http response --- 02:50:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:50:03 DEBUG ----- output ----- 02:50:03 DEBUG { 02:50:03 DEBUG "status": "success", 02:50:03 DEBUG "data": { 02:50:03 DEBUG "resultType": "vector", 02:50:03 DEBUG "result": [ 02:50:03 DEBUG { 02:50:03 DEBUG "metric": { 02:50:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG [...] 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG 02:50:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:50:03 DEBUG }, 02:50:03 DEBUG "value": [ 02:50:03 DEBUG 1712973003, 02:50:03 DEBUG "0" 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG ] 02:50:03 DEBUG } 02:50:03 DEBUG } 02:50:15 INFO 02:50:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:50:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:50:15 INFO [loop_until]: OK (rc = 0) 02:50:15 DEBUG --- stdout --- 02:50:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12022m 8572Mi am-6d96779b5d-qbg7l 9692m 8476Mi am-6d96779b5d-zn69c 11800m 8478Mi ds-cts-0 10620m 13509Mi ds-cts-1 11574m 13554Mi ds-cts-2 10539m 13517Mi ds-idrepo-0 8178m 5594Mi ds-idrepo-1 4272m 5522Mi ds-idrepo-2 4151m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1103Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 7m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5295m 4125Mi 02:50:15 DEBUG --- stderr --- 02:50:15 DEBUG 02:50:21 INFO 02:50:21 INFO [loop_until]: kubectl --namespace=xlou top node 02:50:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:50:21 INFO [loop_until]: OK (rc = 0) 02:50:21 DEBUG --- stdout --- 02:50:21 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4744Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1875Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10242m 34% 10354Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14072m 47% 3112Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12096m 40% 10285Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 4346Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12879m 43% 10415Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4415m 27% 6299Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10678m 67% 14423Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10837m 68% 14491Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4191m 26% 6699Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10824m 68% 14466Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8236m 51% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5390m 33% 7863Mi 13% 02:50:21 DEBUG --- stderr --- 02:50:21 DEBUG 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-idrepo-0" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "1741.5333333333333" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0.023" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973062" 02:51:02 ERROR [http_cmd]: ERROR 02:51:02 ERROR --- status code --- 02:51:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:51:02 ERROR --- http response --- 02:51:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:51:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "instance": "10.142.0.107:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "instance": "10.142.0.77:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "642831844352" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "1733.1" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-cts-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0.0002333333333333333" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "1734.7" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "313675.5" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-cts-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "27.433333333333334" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "instance": "10.142.0.107:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "instance": "10.142.0.77:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "51313297.06666666" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-cts-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "1734.8999999999999" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-cts-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "283258220.2798117" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-idrepo-0" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973062" 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "instance": "10.142.0.107:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "instance": "10.142.0.77:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "51309921" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "pod": "ds-idrepo-1" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "pod": "ds-idrepo-0" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973062" 02:51:02 INFO [http_cmd]: http status code OK 02:51:02 DEBUG --- status code --- 02:51:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:02 DEBUG --- http response --- 02:51:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:02 DEBUG ----- output ----- 02:51:02 DEBUG { 02:51:02 DEBUG "status": "success", 02:51:02 DEBUG "data": { 02:51:02 DEBUG "resultType": "vector", 02:51:02 DEBUG "result": [ 02:51:02 DEBUG { 02:51:02 DEBUG "metric": { 02:51:02 DEBUG "instance": "10.142.0.107:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG [...] 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG 02:51:02 DEBUG "instance": "10.142.0.77:9100" 02:51:02 DEBUG }, 02:51:02 DEBUG "value": [ 02:51:02 DEBUG 1712973062, 02:51:02 DEBUG "0.16146666666665985" 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG ] 02:51:02 DEBUG } 02:51:02 DEBUG } 02:51:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973063" 02:51:03 INFO [http_cmd]: http status code OK 02:51:03 DEBUG --- status code --- 02:51:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:03 DEBUG --- http response --- 02:51:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:03 DEBUG ----- output ----- 02:51:03 DEBUG { 02:51:03 DEBUG "status": "success", 02:51:03 DEBUG "data": { 02:51:03 DEBUG "resultType": "vector", 02:51:03 DEBUG "result": [ 02:51:03 DEBUG { 02:51:03 DEBUG "metric": { 02:51:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG [...] 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 1712973063, 02:51:03 DEBUG "5.2621217763318615" 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG } 02:51:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973063" 02:51:03 INFO [http_cmd]: http status code OK 02:51:03 DEBUG --- status code --- 02:51:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:03 DEBUG --- http response --- 02:51:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:51:03 DEBUG ----- output ----- 02:51:03 DEBUG { 02:51:03 DEBUG "status": "success", 02:51:03 DEBUG "data": { 02:51:03 DEBUG "resultType": "vector", 02:51:03 DEBUG "result": [ 02:51:03 DEBUG { 02:51:03 DEBUG "metric": { 02:51:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG [...] 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 1712973063, 02:51:03 DEBUG "0" 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG } 02:51:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973063" 02:51:03 INFO [http_cmd]: http status code OK 02:51:03 DEBUG --- status code --- 02:51:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:03 DEBUG --- http response --- 02:51:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:03 DEBUG ----- output ----- 02:51:03 DEBUG { 02:51:03 DEBUG "status": "success", 02:51:03 DEBUG "data": { 02:51:03 DEBUG "resultType": "vector", 02:51:03 DEBUG "result": [ 02:51:03 DEBUG { 02:51:03 DEBUG "metric": { 02:51:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG [...] 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 1712973063, 02:51:03 DEBUG "6239313920" 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG } 02:51:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973063" 02:51:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973063" 02:51:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973063" 02:51:03 INFO [http_cmd]: http status code OK 02:51:03 DEBUG --- status code --- 02:51:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:03 DEBUG --- http response --- 02:51:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:51:03 DEBUG ----- output ----- 02:51:03 DEBUG { 02:51:03 DEBUG "status": "success", 02:51:03 DEBUG "data": { 02:51:03 DEBUG "resultType": "vector", 02:51:03 DEBUG "result": [ 02:51:03 DEBUG { 02:51:03 DEBUG "metric": { 02:51:03 DEBUG "instance": "10.142.0.107:9100" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG [...] 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG "instance": "10.142.0.77:9100" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 1712973063, 02:51:03 DEBUG "0.05166666666666657" 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG } 02:51:03 INFO [http_cmd]: http status code OK 02:51:03 DEBUG --- status code --- 02:51:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:03 DEBUG --- http response --- 02:51:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:51:03 DEBUG ----- output ----- 02:51:03 DEBUG { 02:51:03 DEBUG "status": "success", 02:51:03 DEBUG "data": { 02:51:03 DEBUG "resultType": "vector", 02:51:03 DEBUG "result": [ 02:51:03 DEBUG { 02:51:03 DEBUG "metric": { 02:51:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG [...] 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 1712973063, 02:51:03 DEBUG "10.485769631623967" 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG } 02:51:03 INFO [http_cmd]: http status code OK 02:51:03 DEBUG --- status code --- 02:51:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:51:03 DEBUG --- http response --- 02:51:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:51:03 DEBUG ----- output ----- 02:51:03 DEBUG { 02:51:03 DEBUG "status": "success", 02:51:03 DEBUG "data": { 02:51:03 DEBUG "resultType": "vector", 02:51:03 DEBUG "result": [ 02:51:03 DEBUG { 02:51:03 DEBUG "metric": { 02:51:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG [...] 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG 02:51:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:51:03 DEBUG }, 02:51:03 DEBUG "value": [ 02:51:03 DEBUG 1712973063, 02:51:03 DEBUG "0" 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG ] 02:51:03 DEBUG } 02:51:03 DEBUG } 02:51:15 INFO 02:51:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:51:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:51:15 INFO [loop_until]: OK (rc = 0) 02:51:15 DEBUG --- stdout --- 02:51:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11922m 8571Mi am-6d96779b5d-qbg7l 9891m 8476Mi am-6d96779b5d-zn69c 12015m 8484Mi ds-cts-0 10880m 13519Mi ds-cts-1 11185m 13512Mi ds-cts-2 10390m 13519Mi ds-idrepo-0 8178m 5594Mi ds-idrepo-1 4323m 5522Mi ds-idrepo-2 4153m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 5m 1103Mi idm-b85f9d978-dz57p 4m 1115Mi lodemon-8dc49bd56-qr24k 7m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5198m 4122Mi 02:51:15 DEBUG --- stderr --- 02:51:15 DEBUG 02:51:22 INFO 02:51:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:51:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:51:22 INFO [loop_until]: OK (rc = 0) 02:51:22 DEBUG --- stdout --- 02:51:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4734Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 1875Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10239m 34% 10355Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14043m 47% 3088Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11985m 40% 10286Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12661m 42% 10412Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4422m 27% 6297Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10807m 68% 14415Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10974m 69% 14523Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4111m 25% 6698Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10902m 68% 14424Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8283m 52% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5377m 33% 7865Mi 13% 02:51:22 DEBUG --- stderr --- 02:51:22 DEBUG 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-idrepo-0" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "0" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "1737.7" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "0.023" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973122" 02:52:02 ERROR [http_cmd]: ERROR 02:52:02 ERROR --- status code --- 02:52:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:52:02 ERROR --- http response --- 02:52:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:52:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "instance": "10.142.0.107:9100" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "instance": "10.142.0.77:9100" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "645460130816" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-cts-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "0.0001666666666666667" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "1741.1" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG "metric": { 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG { 02:52:02 DEBUG }, 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "value": [ 02:52:02 DEBUG "data": { 02:52:02 DEBUG 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG 02:52:02 DEBUG "result": [ 02:52:02 DEBUG 02:52:02 DEBUG { 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG [...] 02:52:02 DEBUG "metric": { 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG 02:52:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG 02:52:02 DEBUG }, 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG 02:52:02 DEBUG "value": [ 02:52:02 DEBUG "pod": "ds-cts-1" 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG 02:52:02 DEBUG }, 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG 02:52:02 DEBUG "value": [ 02:52:02 DEBUG { 02:52:02 DEBUG 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG [...] 02:52:02 DEBUG "29.133333333333333" 02:52:02 DEBUG "data": { 02:52:02 DEBUG 02:52:02 DEBUG ] 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG 02:52:02 DEBUG } 02:52:02 DEBUG "result": [ 02:52:02 DEBUG 02:52:02 DEBUG ] 02:52:02 DEBUG { 02:52:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:02 DEBUG } 02:52:02 DEBUG "metric": { 02:52:02 DEBUG }, 02:52:02 DEBUG } 02:52:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:02 DEBUG "value": [ 02:52:02 DEBUG }, 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG "313491.5" 02:52:02 DEBUG 02:52:02 DEBUG ] 02:52:02 DEBUG 02:52:02 DEBUG } 02:52:02 DEBUG 02:52:02 DEBUG ] 02:52:02 DEBUG [...] 02:52:02 DEBUG } 02:52:02 DEBUG 02:52:02 DEBUG } 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "1741.7333333333333" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "instance": "10.142.0.107:9100" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "instance": "10.142.0.77:9100" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "51285258.6" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-cts-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "0" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "1741.4333333333334" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "1788.4118695194668" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-cts-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "0" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "9429.80803928446" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973122" 02:52:02 INFO [http_cmd]: http status code OK 02:52:02 DEBUG --- status code --- 02:52:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:02 DEBUG --- http response --- 02:52:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:02 DEBUG ----- output ----- 02:52:02 DEBUG { 02:52:02 DEBUG "status": "success", 02:52:02 DEBUG "data": { 02:52:02 DEBUG "resultType": "vector", 02:52:02 DEBUG "result": [ 02:52:02 DEBUG { 02:52:02 DEBUG "metric": { 02:52:02 DEBUG "pod": "ds-idrepo-1" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG [...] 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG 02:52:02 DEBUG "pod": "ds-idrepo-0" 02:52:02 DEBUG }, 02:52:02 DEBUG "value": [ 02:52:02 DEBUG 1712973122, 02:52:02 DEBUG "0" 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG ] 02:52:02 DEBUG } 02:52:02 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973123" 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "instance": "10.142.0.107:9100" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "instance": "10.142.0.77:9100" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "51282109.06666666" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "pod": "ds-idrepo-1" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "pod": "ds-idrepo-0" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "0" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "instance": "10.142.0.107:9100" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "instance": "10.142.0.77:9100" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "0.17886666666667375" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "5.243769880307406" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "pod": "ds-idrepo-1" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "0" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "6238322688" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973123" 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "pod": "ds-idrepo-1" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "2.343134710200401" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973123" 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "instance": "10.142.0.107:9100" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "instance": "10.142.0.77:9100" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "0.05766666666666751" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:03 INFO [http_cmd]: http status code OK 02:52:03 DEBUG --- status code --- 02:52:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:52:03 DEBUG --- http response --- 02:52:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:52:03 DEBUG ----- output ----- 02:52:03 DEBUG { 02:52:03 DEBUG "status": "success", 02:52:03 DEBUG "data": { 02:52:03 DEBUG "resultType": "vector", 02:52:03 DEBUG "result": [ 02:52:03 DEBUG { 02:52:03 DEBUG "metric": { 02:52:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG [...] 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG 02:52:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:52:03 DEBUG }, 02:52:03 DEBUG "value": [ 02:52:03 DEBUG 1712973123, 02:52:03 DEBUG "0" 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG ] 02:52:03 DEBUG } 02:52:03 DEBUG } 02:52:15 INFO 02:52:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:52:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:52:15 INFO [loop_until]: OK (rc = 0) 02:52:15 DEBUG --- stdout --- 02:52:15 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12177m 8567Mi am-6d96779b5d-qbg7l 9467m 8478Mi am-6d96779b5d-zn69c 11396m 8481Mi ds-cts-0 10542m 13517Mi ds-cts-1 10931m 13512Mi ds-cts-2 10648m 13518Mi ds-idrepo-0 8059m 5594Mi ds-idrepo-1 4270m 5522Mi ds-idrepo-2 4094m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5249m 4122Mi 02:52:15 DEBUG --- stderr --- 02:52:15 DEBUG 02:52:22 INFO 02:52:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:52:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:52:22 INFO [loop_until]: OK (rc = 0) 02:52:22 DEBUG --- stdout --- 02:52:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10240m 34% 10366Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14118m 47% 3086Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12128m 40% 10282Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4335Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12800m 42% 10407Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4431m 27% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10817m 68% 14412Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11128m 70% 14383Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4254m 26% 6695Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10813m 68% 14421Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8289m 52% 6381Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5230m 32% 7860Mi 13% 02:52:22 DEBUG --- stderr --- 02:52:22 DEBUG 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 [35mDEBUG --- http response --- 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "ds-idrepo-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "ds-idrepo-0" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "0" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "1734.5" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "0.023" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973182" 02:53:02 ERROR [http_cmd]: ERROR 02:53:02 ERROR --- status code --- 02:53:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:53:02 ERROR --- http response --- 02:53:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:53:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "instance": "10.142.0.107:9100" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "instance": "10.142.0.77:9100" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "648136193024" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "1741.1666666666667" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "ds-idrepo-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "ds-cts-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "0.0009666666666666667" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "313802" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "1739.8666666666666" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "ds-idrepo-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "ds-cts-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "29.7" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "instance": "10.142.0.107:9100" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "instance": "10.142.0.77:9100" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "51690628.13333333" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973182" 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "ds-idrepo-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "ds-cts-1" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "0" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:02 INFO [http_cmd]: http status code OK 02:53:02 DEBUG --- status code --- 02:53:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:02 DEBUG --- http response --- 02:53:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:02 DEBUG ----- output ----- 02:53:02 DEBUG { 02:53:02 DEBUG "status": "success", 02:53:02 DEBUG "data": { 02:53:02 DEBUG "resultType": "vector", 02:53:02 DEBUG "result": [ 02:53:02 DEBUG { 02:53:02 DEBUG "metric": { 02:53:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG [...] 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG 02:53:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:02 DEBUG }, 02:53:02 DEBUG "value": [ 02:53:02 DEBUG 1712973182, 02:53:02 DEBUG "1739.7" 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG ] 02:53:02 DEBUG } 02:53:02 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-cts-2" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "47461665.41805075" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "ds-idrepo-1" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-cts-1" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-idrepo-1" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "ds-idrepo-1" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-idrepo-0" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973183" 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "instance": "10.142.0.107:9100" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "instance": "10.142.0.77:9100" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "51687363.66666667" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "ds-idrepo-1" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-idrepo-0" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "instance": "10.142.0.107:9100" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "instance": "10.142.0.77:9100" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0.18150000000001257" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "5.284034182815771" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-cts-2" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "35357202.34645558" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "6239498240" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973183" 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "instance": "10.142.0.107:9100" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "instance": "10.142.0.77:9100" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0.057666666666666554" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "ds-idrepo-1" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973183" 02:53:03 INFO [http_cmd]: http status code OK 02:53:03 DEBUG --- status code --- 02:53:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:53:03 DEBUG --- http response --- 02:53:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:53:03 DEBUG ----- output ----- 02:53:03 DEBUG { 02:53:03 DEBUG "status": "success", 02:53:03 DEBUG "data": { 02:53:03 DEBUG "resultType": "vector", 02:53:03 DEBUG "result": [ 02:53:03 DEBUG { 02:53:03 DEBUG "metric": { 02:53:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG [...] 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG 02:53:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:53:03 DEBUG }, 02:53:03 DEBUG "value": [ 02:53:03 DEBUG 1712973183, 02:53:03 DEBUG "0" 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG ] 02:53:03 DEBUG } 02:53:03 DEBUG } 02:53:15 INFO 02:53:15 INFO [loop_until]: kubectl --namespace=xlou top pods 02:53:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:53:16 INFO [loop_until]: OK (rc = 0) 02:53:16 DEBUG --- stdout --- 02:53:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11983m 8572Mi am-6d96779b5d-qbg7l 9633m 8475Mi am-6d96779b5d-zn69c 11582m 8480Mi ds-cts-0 10745m 13646Mi ds-cts-1 10618m 13514Mi ds-cts-2 10573m 13519Mi ds-idrepo-0 8058m 5594Mi ds-idrepo-1 4270m 5523Mi ds-idrepo-2 4153m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1120Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 8m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5344m 4123Mi 02:53:16 DEBUG --- stderr --- 02:53:16 DEBUG 02:53:22 INFO 02:53:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:53:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:53:22 INFO [loop_until]: OK (rc = 0) 02:53:22 DEBUG --- stdout --- 02:53:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4743Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1874Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10379m 34% 10356Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14044m 47% 3105Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12121m 40% 10291Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 4331Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12755m 42% 10405Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4366m 27% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10851m 68% 14408Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11126m 70% 14491Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4287m 26% 6696Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10841m 68% 14416Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8250m 51% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5375m 33% 7861Mi 13% 02:53:22 DEBUG --- stderr --- 02:53:22 DEBUG 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "ds-idrepo-1" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "ds-idrepo-0" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "0" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "1734.1" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973242" 02:54:02 ERROR [http_cmd]: ERROR 02:54:02 ERROR --- status code --- 02:54:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:54:02 ERROR --- http response --- 02:54:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:54:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "0.023" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "1745.9333333333334" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "instance": "10.142.0.107:9100" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "instance": "10.142.0.77:9100" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "650913459200" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "313783.5" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "ds-idrepo-1" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "ds-cts-1" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "0.0015468472222222222" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "1747.1" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "pod": "ds-idrepo-1" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "pod": "ds-cts-1" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "28.966666666666665" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973242" 02:54:02 INFO [http_cmd]: http status code OK 02:54:02 DEBUG --- status code --- 02:54:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:02 DEBUG --- http response --- 02:54:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:02 DEBUG ----- output ----- 02:54:02 DEBUG { 02:54:02 DEBUG "status": "success", 02:54:02 DEBUG "data": { 02:54:02 DEBUG "resultType": "vector", 02:54:02 DEBUG "result": [ 02:54:02 DEBUG { 02:54:02 DEBUG "metric": { 02:54:02 DEBUG "instance": "10.142.0.107:9100" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG [...] 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG 02:54:02 DEBUG "instance": "10.142.0.77:9100" 02:54:02 DEBUG }, 02:54:02 DEBUG "value": [ 02:54:02 DEBUG 1712973242, 02:54:02 DEBUG "51423288.266666666" 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG ] 02:54:02 DEBUG } 02:54:02 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-cts-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "1747.1666666666667" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG { 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "data": { 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG }, 02:54:03 DEBUG "result": [ 02:54:03 DEBUG "value": [ 02:54:03 DEBUG { 02:54:03 DEBUG 02:54:03 DEBUG "metric": { 02:54:03 DEBUG 02:54:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:54:03 DEBUG 02:54:03 DEBUG }, 02:54:03 DEBUG [...] 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-cts-1" 02:54:03 DEBUG [...] 02:54:03 DEBUG }, 02:54:03 DEBUG 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG 02:54:03 DEBUG "0" 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG ] 02:54:03 DEBUG }, 02:54:03 DEBUG } 02:54:03 DEBUG "value": [ 02:54:03 DEBUG ] 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG } 02:54:03 DEBUG "2982.1151593808363" 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "instance": "10.142.0.107:9100" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "instance": "10.142.0.77:9100" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "51419661.2" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "1037.2574467411605" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-idrepo-0" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-idrepo-0" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "instance": "10.142.0.107:9100" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "instance": "10.142.0.77:9100" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0.17796666666666613" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "5.2512080001413795" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "6239592448" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973243" 02:54:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973243" 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "ds-idrepo-1" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "4.270957057959608" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG --- http response --- 02:54:03 INFO [http_cmd]: http status code OK 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:54:03 DEBUG --- status code --- 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:54:03 DEBUG { 02:54:03 DEBUG --- http response --- 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "instance": "10.142.0.107:9100" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "instance": "10.142.0.77:9100" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0.05533333333333274" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:54:03 DEBUG ----- output ----- 02:54:03 DEBUG { 02:54:03 DEBUG "status": "success", 02:54:03 DEBUG "data": { 02:54:03 DEBUG "resultType": "vector", 02:54:03 DEBUG "result": [ 02:54:03 DEBUG { 02:54:03 DEBUG "metric": { 02:54:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG [...] 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG 02:54:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:54:03 DEBUG }, 02:54:03 DEBUG "value": [ 02:54:03 DEBUG 1712973243, 02:54:03 DEBUG "0" 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG ] 02:54:03 DEBUG } 02:54:03 DEBUG } 02:54:16 INFO 02:54:16 INFO [loop_until]: kubectl --namespace=xlou top pods 02:54:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:54:16 INFO [loop_until]: OK (rc = 0) 02:54:16 DEBUG --- stdout --- 02:54:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11847m 8566Mi am-6d96779b5d-qbg7l 10227m 8481Mi am-6d96779b5d-zn69c 11713m 8481Mi ds-cts-0 10896m 13683Mi ds-cts-1 10821m 13515Mi ds-cts-2 10534m 13519Mi ds-idrepo-0 8144m 5593Mi ds-idrepo-1 4308m 5522Mi ds-idrepo-2 4132m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 7m 1116Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5372m 4124Mi 02:54:16 DEBUG --- stderr --- 02:54:16 DEBUG 02:54:22 INFO 02:54:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:54:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:54:22 INFO [loop_until]: OK (rc = 0) 02:54:22 DEBUG --- stdout --- 02:54:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4733Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10268m 34% 10362Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14109m 47% 3106Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12147m 40% 10281Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4337Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12912m 43% 10413Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4380m 27% 6302Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10422m 65% 14432Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11034m 69% 14418Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4235m 26% 6697Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10531m 66% 14420Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8236m 51% 6380Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5407m 34% 7863Mi 13% 02:54:22 DEBUG --- stderr --- 02:54:22 DEBUG 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "ds-idrepo-1" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "ds-idrepo-0" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "0" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "1741.9666666666667" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973302" 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 ERROR [http_cmd]: ERROR 02:55:02 ERROR --- status code --- 02:55:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:55:02 ERROR --- http response --- 02:55:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:55:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "0.023" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973302" 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "1751.4333333333334" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "instance": "10.142.0.107:9100" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "instance": "10.142.0.77:9100" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "653561537536" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "314186.5" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "ds-idrepo-1" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "ds-cts-1" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "0.0005692666666666667" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "ds-idrepo-1" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "ds-cts-1" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "28.8" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973302" 02:55:02 INFO [http_cmd]: http status code OK 02:55:02 DEBUG --- status code --- 02:55:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:02 DEBUG --- http response --- 02:55:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:02 DEBUG ----- output ----- 02:55:02 DEBUG { 02:55:02 DEBUG "status": "success", 02:55:02 DEBUG "data": { 02:55:02 DEBUG "resultType": "vector", 02:55:02 DEBUG "result": [ 02:55:02 DEBUG { 02:55:02 DEBUG "metric": { 02:55:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG [...] 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG 02:55:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:02 DEBUG }, 02:55:02 DEBUG "value": [ 02:55:02 DEBUG 1712973302, 02:55:02 DEBUG "1749.9333333333334" 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG ] 02:55:02 DEBUG } 02:55:02 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "instance": "10.142.0.107:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "instance": "10.142.0.77:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "51512098.9" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "ds-idrepo-1" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-cts-1" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-cts-2" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "482843599.1359242" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "1749.6333333333332" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-cts-2" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "22178571.844962634" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "ds-idrepo-1" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-cts-1" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "instance": "10.142.0.107:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "instance": "10.142.0.77:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "51508994.733333334" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "ds-idrepo-1" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-idrepo-0" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "instance": "10.142.0.107:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "instance": "10.142.0.77:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0.1650333333333336" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "ds-idrepo-1" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-idrepo-0" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "5.278883677471965" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-cts-2" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "34069850.08208478" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "6238629888" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "ds-cts-2" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "6.474266891252949" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "instance": "10.142.0.107:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "instance": "10.142.0.77:9100" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0.04799999999999898" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973303" 02:55:03 INFO [http_cmd]: http status code OK 02:55:03 DEBUG --- status code --- 02:55:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:55:03 DEBUG --- http response --- 02:55:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:55:03 DEBUG ----- output ----- 02:55:03 DEBUG { 02:55:03 DEBUG "status": "success", 02:55:03 DEBUG "data": { 02:55:03 DEBUG "resultType": "vector", 02:55:03 DEBUG "result": [ 02:55:03 DEBUG { 02:55:03 DEBUG "metric": { 02:55:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG [...] 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG 02:55:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:55:03 DEBUG }, 02:55:03 DEBUG "value": [ 02:55:03 DEBUG 1712973303, 02:55:03 DEBUG "0" 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG ] 02:55:03 DEBUG } 02:55:03 DEBUG } 02:55:16 INFO 02:55:16 INFO [loop_until]: kubectl --namespace=xlou top pods 02:55:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:55:16 INFO [loop_until]: OK (rc = 0) 02:55:16 DEBUG --- stdout --- 02:55:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12012m 8566Mi am-6d96779b5d-qbg7l 9832m 8482Mi am-6d96779b5d-zn69c 11899m 8480Mi ds-cts-0 10424m 13517Mi ds-cts-1 10995m 13568Mi ds-cts-2 10967m 13518Mi ds-idrepo-0 8039m 5594Mi ds-idrepo-1 4370m 5522Mi ds-idrepo-2 4269m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5255m 4123Mi 02:55:16 DEBUG --- stderr --- 02:55:16 DEBUG 02:55:22 INFO 02:55:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:55:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:55:22 INFO [loop_until]: OK (rc = 0) 02:55:22 DEBUG --- stdout --- 02:55:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4749Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1857Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10441m 34% 10362Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14059m 47% 3102Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12018m 40% 10290Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12777m 42% 10409Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4416m 27% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10822m 68% 14442Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11117m 69% 14415Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4236m 26% 6699Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10811m 68% 14420Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8259m 51% 6386Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5391m 33% 7863Mi 13% 02:55:22 DEBUG --- stderr --- 02:55:22 DEBUG 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:02 DEBUG ----- output ----- 02:56:02 DEBUG { 02:56:02 DEBUG "status": "success", 02:56:02 DEBUG "data": { 02:56:02 DEBUG "resultType": "vector", 02:56:02 DEBUG "result": [ 02:56:02 DEBUG { 02:56:02 DEBUG "metric": { 02:56:02 DEBUG "pod": "ds-idrepo-1" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG [...] 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG "pod": "ds-idrepo-0" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 1712973362, 02:56:02 DEBUG "0" 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG } 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:02 DEBUG ----- output ----- 02:56:02 DEBUG { 02:56:02 DEBUG "status": "success", 02:56:02 DEBUG "data": { 02:56:02 DEBUG "resultType": "vector", 02:56:02 DEBUG "result": [ 02:56:02 DEBUG { 02:56:02 DEBUG "metric": { 02:56:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG [...] 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 1712973362, 02:56:02 DEBUG "1762.7333333333333" 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG } 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973362" 02:56:02 ERROR [http_cmd]: ERROR 02:56:02 ERROR --- status code --- 02:56:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:56:02 ERROR --- http response --- 02:56:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:56:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973362" 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:02 DEBUG ----- output ----- 02:56:02 DEBUG { 02:56:02 DEBUG "status": "success", 02:56:02 DEBUG "data": { 02:56:02 DEBUG "resultType": "vector", 02:56:02 DEBUG "result": [ 02:56:02 DEBUG { 02:56:02 DEBUG "metric": { 02:56:02 DEBUG "instance": "10.142.0.107:9100" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG [...] 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG "instance": "10.142.0.77:9100" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 1712973362, 02:56:02 DEBUG "656091034624" 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG } 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:02 DEBUG ----- output ----- 02:56:02 DEBUG { 02:56:02 DEBUG "status": "success", 02:56:02 DEBUG "data": { 02:56:02 DEBUG "resultType": "vector", 02:56:02 DEBUG "result": [ 02:56:02 DEBUG { 02:56:02 DEBUG "metric": { 02:56:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG [...] 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 1712973362, 02:56:02 DEBUG "0.023" 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG } 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:02 DEBUG ----- output ----- 02:56:02 DEBUG { 02:56:02 DEBUG "status": "success", 02:56:02 DEBUG "data": { 02:56:02 DEBUG "resultType": "vector", 02:56:02 DEBUG "result": [ 02:56:02 DEBUG { 02:56:02 DEBUG "metric": { 02:56:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG [...] 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 1712973362, 02:56:02 DEBUG "314363.5" 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG } 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: http status code OK 02:56:02 DEBUG --- status code --- 02:56:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:02 DEBUG --- http response --- 02:56:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:02 DEBUG ----- output ----- 02:56:02 DEBUG { 02:56:02 DEBUG "status": "success", 02:56:02 DEBUG "data": { 02:56:02 DEBUG "resultType": "vector", 02:56:02 DEBUG "result": [ 02:56:02 DEBUG { 02:56:02 DEBUG "metric": { 02:56:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG [...] 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG 02:56:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:02 DEBUG }, 02:56:02 DEBUG "value": [ 02:56:02 DEBUG 1712973362, 02:56:02 DEBUG "1747.2666666666667" 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG ] 02:56:02 DEBUG } 02:56:02 DEBUG } 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973362" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "ds-idrepo-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973362, 02:56:03 DEBUG "0.00010000000000000009" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "ds-idrepo-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973362, 02:56:03 DEBUG "29.333333333333332" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "1753.2" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "instance": "10.142.0.107:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "instance": "10.142.0.77:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "51479422.46666667" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "ds-idrepo-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-2" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "40214.20921099607" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "1753.1" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-2" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "40354300.89253838" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973363" 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "instance": "10.142.0.107:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "instance": "10.142.0.77:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "51475806.3" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "ds-idrepo-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "5.274962116913763" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "ds-idrepo-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-idrepo-0" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "instance": "10.142.0.107:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "instance": "10.142.0.77:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0.15060000000000237" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "ds-idrepo-1" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-idrepo-0" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-2" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "27018830.417707965" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "6239088640" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "ds-cts-2" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "10.721529055766576" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973363" 02:56:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973363" 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "instance": "10.142.0.107:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "instance": "10.142.0.77:9100" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0.04700000000000012" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:03 INFO [http_cmd]: http status code OK 02:56:03 DEBUG --- status code --- 02:56:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:56:03 DEBUG --- http response --- 02:56:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:56:03 DEBUG ----- output ----- 02:56:03 DEBUG { 02:56:03 DEBUG "status": "success", 02:56:03 DEBUG "data": { 02:56:03 DEBUG "resultType": "vector", 02:56:03 DEBUG "result": [ 02:56:03 DEBUG { 02:56:03 DEBUG "metric": { 02:56:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG [...] 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG 02:56:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:56:03 DEBUG }, 02:56:03 DEBUG "value": [ 02:56:03 DEBUG 1712973363, 02:56:03 DEBUG "0" 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG ] 02:56:03 DEBUG } 02:56:03 DEBUG } 02:56:16 INFO 02:56:16 INFO [loop_until]: kubectl --namespace=xlou top pods 02:56:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:56:16 INFO [loop_until]: OK (rc = 0) 02:56:16 DEBUG --- stdout --- 02:56:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12105m 8566Mi am-6d96779b5d-qbg7l 9800m 8482Mi am-6d96779b5d-zn69c 11721m 8473Mi ds-cts-0 10564m 13518Mi ds-cts-1 10875m 13565Mi ds-cts-2 10689m 13518Mi ds-idrepo-0 7939m 5594Mi ds-idrepo-1 4326m 5522Mi ds-idrepo-2 4163m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5295m 4123Mi 02:56:16 DEBUG --- stderr --- 02:56:16 DEBUG 02:56:22 INFO 02:56:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:56:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:56:22 INFO [loop_until]: OK (rc = 0) 02:56:22 DEBUG --- stdout --- 02:56:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 1857Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10211m 34% 10356Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14059m 47% 3097Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12244m 41% 10283Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4340Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12823m 42% 10409Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4436m 27% 6304Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10901m 68% 14433Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10873m 68% 14375Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4265m 26% 6701Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10822m 68% 14419Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8297m 52% 6386Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5399m 33% 7864Mi 13% 02:56:22 DEBUG --- stderr --- 02:56:22 DEBUG 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973422" 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973422" 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:02 DEBUG ----- output ----- 02:57:02 DEBUG { 02:57:02 DEBUG "status": "success", 02:57:02 DEBUG "data": { 02:57:02 DEBUG "resultType": "vector", 02:57:02 DEBUG "result": [ 02:57:02 DEBUG { 02:57:02 DEBUG "metric": { 02:57:02 DEBUG "pod": "ds-idrepo-1" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG [...] 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG "pod": "ds-idrepo-0" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 1712973422, 02:57:02 DEBUG "0" 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG } 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973422" 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:02 DEBUG ----- output ----- 02:57:02 DEBUG { 02:57:02 DEBUG "status": "success", 02:57:02 DEBUG "data": { 02:57:02 DEBUG "resultType": "vector", 02:57:02 DEBUG "result": [ 02:57:02 DEBUG { 02:57:02 DEBUG "metric": { 02:57:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG [...] 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 1712973422, 02:57:02 DEBUG "1747.7333333333333" 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG } 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973422" 02:57:02 ERROR [http_cmd]: ERROR 02:57:02 ERROR --- status code --- 02:57:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:57:02 ERROR --- http response --- 02:57:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:57:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973422" 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973422" 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:02 DEBUG ----- output ----- 02:57:02 DEBUG { 02:57:02 DEBUG "status": "success", 02:57:02 DEBUG "data": { 02:57:02 DEBUG "resultType": "vector", 02:57:02 DEBUG "result": [ 02:57:02 DEBUG { 02:57:02 DEBUG "metric": { 02:57:02 DEBUG "instance": "10.142.0.107:9100" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG [...] 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG "instance": "10.142.0.77:9100" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 1712973422, 02:57:02 DEBUG "658489979904" 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG } 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:02 DEBUG ----- output ----- 02:57:02 DEBUG { 02:57:02 DEBUG "status": "success", 02:57:02 DEBUG "data": { 02:57:02 DEBUG "resultType": "vector", 02:57:02 DEBUG "result": [ 02:57:02 DEBUG { 02:57:02 DEBUG "metric": { 02:57:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG [...] 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 1712973422, 02:57:02 DEBUG "0.023" 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG } 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973422" 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:02 DEBUG ----- output ----- 02:57:02 DEBUG { 02:57:02 DEBUG "status": "success", 02:57:02 DEBUG "data": { 02:57:02 DEBUG "resultType": "vector", 02:57:02 DEBUG "result": [ 02:57:02 DEBUG { 02:57:02 DEBUG "metric": { 02:57:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG [...] 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 1712973422, 02:57:02 DEBUG "314316.5" 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG } 02:57:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973422" 02:57:02 INFO [http_cmd]: http status code OK 02:57:02 DEBUG --- status code --- 02:57:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:02 DEBUG --- http response --- 02:57:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:02 DEBUG ----- output ----- 02:57:02 DEBUG { 02:57:02 DEBUG "status": "success", 02:57:02 DEBUG "data": { 02:57:02 DEBUG "resultType": "vector", 02:57:02 DEBUG "result": [ 02:57:02 DEBUG { 02:57:02 DEBUG "metric": { 02:57:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG [...] 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG 02:57:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:02 DEBUG }, 02:57:02 DEBUG "value": [ 02:57:02 DEBUG 1712973422, 02:57:02 DEBUG "1740.8666666666666" 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG ] 02:57:02 DEBUG } 02:57:02 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "ds-idrepo-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "ds-cts-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "29.566666666666666" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "ds-idrepo-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "ds-cts-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0.00012282500000000002" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "1742.5" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "instance": "10.142.0.107:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "instance": "10.142.0.77:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "51094195.766666666" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "ds-idrepo-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "ds-cts-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "1741.4333333333334" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "305940832.8402392" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973423" 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "instance": "10.142.0.107:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "instance": "10.142.0.77:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "51093896.06666666" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973423" 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973423" 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "instance": "10.142.0.107:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "instance": "10.142.0.77:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0.10616666666665774" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG "result": [ 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG { 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "5.259490105628871" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "ds-idrepo-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "ds-cts-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "ds-idrepo-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "ds-idrepo-0" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "ds-idrepo-1" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "ds-idrepo-0" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "6241480704" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "10.63018512584538" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "instance": "10.142.0.107:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "instance": "10.142.0.77:9100" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0.03333333333333522" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973423" 02:57:03 INFO [http_cmd]: http status code OK 02:57:03 DEBUG --- status code --- 02:57:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:57:03 DEBUG --- http response --- 02:57:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:57:03 DEBUG ----- output ----- 02:57:03 DEBUG { 02:57:03 DEBUG "status": "success", 02:57:03 DEBUG "data": { 02:57:03 DEBUG "resultType": "vector", 02:57:03 DEBUG "result": [ 02:57:03 DEBUG { 02:57:03 DEBUG "metric": { 02:57:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG [...] 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG 02:57:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:57:03 DEBUG }, 02:57:03 DEBUG "value": [ 02:57:03 DEBUG 1712973423, 02:57:03 DEBUG "0" 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG ] 02:57:03 DEBUG } 02:57:03 DEBUG } 02:57:16 INFO 02:57:16 INFO [loop_until]: kubectl --namespace=xlou top pods 02:57:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:57:16 INFO [loop_until]: OK (rc = 0) 02:57:16 DEBUG --- stdout --- 02:57:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12005m 8568Mi am-6d96779b5d-qbg7l 9799m 8480Mi am-6d96779b5d-zn69c 11800m 8477Mi ds-cts-0 10477m 13518Mi ds-cts-1 10923m 13603Mi ds-cts-2 10433m 13558Mi ds-idrepo-0 8047m 5594Mi ds-idrepo-1 4276m 5523Mi ds-idrepo-2 4159m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5174m 4124Mi 02:57:16 DEBUG --- stderr --- 02:57:16 DEBUG 02:57:22 INFO 02:57:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:57:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:57:22 INFO [loop_until]: OK (rc = 0) 02:57:22 DEBUG --- stdout --- 02:57:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1857Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10233m 34% 10366Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13990m 46% 3096Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11969m 40% 10284Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 128m 0% 4360Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12809m 42% 10415Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4394m 27% 6308Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10890m 68% 14422Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11321m 71% 14399Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4216m 26% 6696Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10798m 67% 14479Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8225m 51% 6384Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5347m 33% 7865Mi 13% 02:57:22 DEBUG --- stderr --- 02:57:22 DEBUG 02:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973482" 02:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973482" 02:58:02 INFO [http_cmd]: http status code OK 02:58:02 INFO [http_cmd]: http status code OK 02:58:02 DEBUG --- status code --- 02:58:02 DEBUG --- status code --- 02:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:02 DEBUG --- http response --- 02:58:02 DEBUG --- http response --- 02:58:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:02 DEBUG ----- output ----- 02:58:02 DEBUG { 02:58:02 DEBUG "status": "success", 02:58:02 DEBUG "data": { 02:58:02 DEBUG "resultType": "vector", 02:58:02 DEBUG "result": [ 02:58:02 DEBUG { 02:58:02 DEBUG "metric": { 02:58:02 DEBUG "pod": "ds-idrepo-1" 02:58:02 DEBUG }, 02:58:02 DEBUG "value": [ 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG [...] 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG "pod": "ds-idrepo-0" 02:58:02 DEBUG }, 02:58:02 DEBUG "value": [ 02:58:02 DEBUG 1712973482, 02:58:02 DEBUG "0" 02:58:02 DEBUG ] 02:58:02 DEBUG } 02:58:02 DEBUG ] 02:58:02 DEBUG } 02:58:02 DEBUG } 02:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973482" 02:58:02 INFO [http_cmd]: http status code OK 02:58:02 DEBUG --- status code --- 02:58:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:02 DEBUG --- http response --- 02:58:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:02 DEBUG ----- output ----- 02:58:02 DEBUG { 02:58:02 DEBUG "status": "success", 02:58:02 DEBUG "data": { 02:58:02 DEBUG "resultType": "vector", 02:58:02 DEBUG "result": [ 02:58:02 DEBUG { 02:58:02 DEBUG "metric": { 02:58:02 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:02 DEBUG }, 02:58:02 DEBUG "value": [ 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG [...] 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG 02:58:02 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:02 DEBUG }, 02:58:02 DEBUG "value": [ 02:58:02 DEBUG 1712973482, 02:58:02 DEBUG "1746.6666666666667" 02:58:02 DEBUG ] 02:58:02 DEBUG } 02:58:02 DEBUG ] 02:58:02 DEBUG } 02:58:02 DEBUG } 02:58:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973482" 02:58:02 ERROR [http_cmd]: ERROR 02:58:02 ERROR --- status code --- 02:58:02 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:58:02 ERROR --- http response --- 02:58:02 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:58:02 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:03 DEBUG }, 02:58:03 DEBUG "instance": "10.142.0.107:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:03 DEBUG 02:58:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:03 DEBUG }, 02:58:03 DEBUG "instance": "10.142.0.77:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0.023" 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "314290.5" 02:58:03 DEBUG ] 02:58:03 DEBUG "661045838848" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "1746.0333333333333" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "ds-idrepo-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "ds-cts-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "28.3" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "ds-idrepo-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "ds-cts-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0.0011648416666666665" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "1742.5333333333333" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "instance": "10.142.0.107:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "instance": "10.142.0.77:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "50994928.06666666" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "ds-idrepo-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "ds-cts-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "1741.8999999999999" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG { 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "ds-idrepo-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "ds-cts-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "196661.67145667062" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "instance": "10.142.0.107:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "instance": "10.142.0.77:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "50991104.56666666" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "instance": "10.142.0.107:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "instance": "10.142.0.77:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0.17493333333333963" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "5.213301009590147" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "ds-idrepo-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "ds-idrepo-0" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "ds-idrepo-1" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "ds-idrepo-0" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "6239924224" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "10.483668176547514" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973483" 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:03 INFO [http_cmd]: http status code OK 02:58:03 DEBUG --- status code --- 02:58:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:58:03 DEBUG --- http response --- 02:58:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:58:03 DEBUG ----- output ----- 02:58:03 DEBUG { 02:58:03 DEBUG "status": "success", 02:58:03 DEBUG "data": { 02:58:03 DEBUG "resultType": "vector", 02:58:03 DEBUG "result": [ 02:58:03 DEBUG { 02:58:03 DEBUG "metric": { 02:58:03 DEBUG "instance": "10.142.0.107:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG [...] 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG 02:58:03 DEBUG "instance": "10.142.0.77:9100" 02:58:03 DEBUG }, 02:58:03 DEBUG "value": [ 02:58:03 DEBUG 1712973483, 02:58:03 DEBUG "0.060666666666666196" 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG ] 02:58:03 DEBUG } 02:58:03 DEBUG } 02:58:16 INFO 02:58:16 INFO [loop_until]: kubectl --namespace=xlou top pods 02:58:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:58:16 INFO [loop_until]: OK (rc = 0) 02:58:16 DEBUG --- stdout --- 02:58:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11850m 8571Mi am-6d96779b5d-qbg7l 9773m 8482Mi am-6d96779b5d-zn69c 11502m 8476Mi ds-cts-0 10476m 13738Mi ds-cts-1 10879m 13525Mi ds-cts-2 10741m 13594Mi ds-idrepo-0 8281m 5594Mi ds-idrepo-1 4222m 5523Mi ds-idrepo-2 4050m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1120Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5301m 4124Mi 02:58:16 DEBUG --- stderr --- 02:58:16 DEBUG 02:58:22 INFO 02:58:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:58:22 INFO [loop_until]: OK (rc = 0) 02:58:22 DEBUG --- stdout --- 02:58:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4728Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 1870Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10285m 34% 10360Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14356m 48% 3133Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12323m 41% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 13086m 43% 10426Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4280m 26% 6301Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10773m 67% 14431Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11232m 70% 14428Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4280m 26% 6699Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10883m 68% 14487Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8265m 52% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5467m 34% 7934Mi 13% 02:58:22 DEBUG --- stderr --- 02:58:22 DEBUG 02:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973542" 02:59:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973542" 02:59:02 INFO [http_cmd]: http status code OK 02:59:02 INFO [http_cmd]: http status code OK 02:59:02 DEBUG --- status code --- 02:59:02 DEBUG --- status code --- 02:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:02 DEBUG --- http response --- 02:59:02 DEBUG --- http response --- 02:59:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 02:59:02 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:02 DEBUG ----- output ----- 02:59:02 DEBUG { 02:59:02 DEBUG "status": "success", 02:59:02 DEBUG "data": { 02:59:02 DEBUG "resultType": "vector", 02:59:02 DEBUG "result": [ 02:59:02 DEBUG { 02:59:02 DEBUG "metric": { 02:59:02 DEBUG "pod": "ds-idrepo-1" 02:59:02 DEBUG }, 02:59:02 DEBUG "value": [ 02:59:02 DEBUG 02:59:02 DEBUG 02:59:02 DEBUG 02:59:02 DEBUG [...] 02:59:02 DEBUG 02:59:02 DEBUG 02:59:02 DEBUG 02:59:02 DEBUG "pod": "ds-idrepo-0" 02:59:02 DEBUG }, 02:59:02 DEBUG "value": [ 02:59:02 DEBUG 1712973542, 02:59:02 DEBUG "0" 02:59:02 DEBUG ] 02:59:02 DEBUG } 02:59:02 DEBUG ] 02:59:02 DEBUG } 02:59:02 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "1763.7333333333333" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973543" 02:59:03 ERROR [http_cmd]: ERROR 02:59:03 ERROR --- status code --- 02:59:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 02:59:03 ERROR --- http response --- 02:59:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 02:59:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0.023" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "instance": "10.142.0.107:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "instance": "10.142.0.77:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "663767428096" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "315024" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "1764.3333333333333" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG { 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "1770.3" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "ds-idrepo-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "ds-cts-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "27.9" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "ds-idrepo-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "ds-cts-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0.00013333333333333334" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "instance": "10.142.0.107:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "instance": "10.142.0.77:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "51297733.43333334" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "ds-idrepo-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "ds-cts-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "instance": "10.142.0.107:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "instance": "10.142.0.77:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0.2036666666666624" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG "pod": "ds-idrepo-1" 02:59:03 DEBUG }, 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG "pod": "ds-cts-1" 02:59:03 DEBUG }, 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "1770" 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "instance": "10.142.0.107:9100" 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:59:03 DEBUG }, 02:59:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "instance": "10.142.0.77:9100" 02:59:03 DEBUG 02:59:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:59:03 DEBUG }, 02:59:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "51293935.766666666" 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG "270413463.4864718" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "5.351182200093919" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "ds-idrepo-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "ds-idrepo-0" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "ds-idrepo-1" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "ds-idrepo-0" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973543" 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "6239666176" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "instance": "10.142.0.107:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "instance": "10.142.0.77:9100" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0.07066666666666632" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973543" 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "10.428606683167354" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:03 INFO [http_cmd]: http status code OK 02:59:03 DEBUG --- status code --- 02:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 02:59:03 DEBUG --- http response --- 02:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 02:59:03 DEBUG ----- output ----- 02:59:03 DEBUG { 02:59:03 DEBUG "status": "success", 02:59:03 DEBUG "data": { 02:59:03 DEBUG "resultType": "vector", 02:59:03 DEBUG "result": [ 02:59:03 DEBUG { 02:59:03 DEBUG "metric": { 02:59:03 DEBUG "pod": "am-6d96779b5d-zn69c" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG [...] 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG 02:59:03 DEBUG "pod": "am-6d96779b5d-pfljh" 02:59:03 DEBUG }, 02:59:03 DEBUG "value": [ 02:59:03 DEBUG 1712973543, 02:59:03 DEBUG "0" 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG ] 02:59:03 DEBUG } 02:59:03 DEBUG } 02:59:16 INFO 02:59:16 INFO [loop_until]: kubectl --namespace=xlou top pods 02:59:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:59:16 INFO [loop_until]: OK (rc = 0) 02:59:16 DEBUG --- stdout --- 02:59:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12474m 8569Mi am-6d96779b5d-qbg7l 9877m 8482Mi am-6d96779b5d-zn69c 11954m 8483Mi ds-cts-0 11370m 13516Mi ds-cts-1 11230m 13511Mi ds-cts-2 10845m 13517Mi ds-idrepo-0 8212m 5594Mi ds-idrepo-1 4301m 5523Mi ds-idrepo-2 4195m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1116Mi lodemon-8dc49bd56-qr24k 6m 67Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5231m 4193Mi 02:59:16 DEBUG --- stderr --- 02:59:16 DEBUG 02:59:22 INFO 02:59:22 INFO [loop_until]: kubectl --namespace=xlou top node 02:59:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 02:59:22 INFO [loop_until]: OK (rc = 0) 02:59:22 DEBUG --- stdout --- 02:59:22 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4734Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1860Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10197m 34% 10353Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13989m 46% 3096Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12040m 40% 10290Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12757m 42% 10415Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4367m 27% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10697m 67% 14455Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11103m 69% 14394Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4208m 26% 6701Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10702m 67% 14435Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8223m 51% 6387Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5346m 33% 7936Mi 13% 02:59:22 DEBUG --- stderr --- 02:59:22 DEBUG 03:00:02 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973602" 03:00:02 INFO [http_cmd]: http status code OK 03:00:02 DEBUG --- status code --- 03:00:02 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:02 DEBUG --- http response --- 03:00:02 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "ds-idrepo-0" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "1734.5" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973603" 03:00:03 ERROR [http_cmd]: ERROR 03:00:03 ERROR --- status code --- 03:00:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:00:03 ERROR --- http response --- 03:00:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:00:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "315049.5" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "instance": "10.142.0.107:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "instance": "10.142.0.77:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "666343132160" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0.023" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "1740.2333333333333" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG }, 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG "pod": "ds-cts-1" 03:00:03 DEBUG }, 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "1739.5333333333333" 03:00:03 DEBUG "25.5" 03:00:03 DEBUG ] 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "ds-cts-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0.0005666666666666667" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "instance": "10.142.0.107:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "instance": "10.142.0.77:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "51619645.56666666" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "instance": "10.142.0.107:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "instance": "10.142.0.77:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0.08933333333332598" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "ds-cts-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "instance": "10.142.0.107:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "instance": "10.142.0.77:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "51616590.266666666" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG { 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:00:03 DEBUG "data": { 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG }, 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "value": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG }, 03:00:03 DEBUG 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG [...] 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG "pod": "ds-cts-1" 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG }, 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG 03:00:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:00:03 DEBUG "0" 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG }, 03:00:03 DEBUG ] 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG } 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG ] 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "340241351.69904065" 03:00:03 DEBUG } 03:00:03 DEBUG "1739.6" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "5.223736504229493" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "ds-idrepo-0" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "ds-idrepo-1" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "ds-idrepo-0" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "instance": "10.142.0.107:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "instance": "10.142.0.77:9100" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0.03299999999999982" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "8.789568702667792" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "6238523392" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973603" 03:00:03 INFO [http_cmd]: http status code OK 03:00:03 DEBUG --- status code --- 03:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:00:03 DEBUG --- http response --- 03:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:00:03 DEBUG ----- output ----- 03:00:03 DEBUG { 03:00:03 DEBUG "status": "success", 03:00:03 DEBUG "data": { 03:00:03 DEBUG "resultType": "vector", 03:00:03 DEBUG "result": [ 03:00:03 DEBUG { 03:00:03 DEBUG "metric": { 03:00:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG [...] 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG 03:00:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:00:03 DEBUG }, 03:00:03 DEBUG "value": [ 03:00:03 DEBUG 1712973603, 03:00:03 DEBUG "0" 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG ] 03:00:03 DEBUG } 03:00:03 DEBUG } 03:00:16 INFO 03:00:16 INFO [loop_until]: kubectl --namespace=xlou top pods 03:00:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:00:16 INFO [loop_until]: OK (rc = 0) 03:00:16 DEBUG --- stdout --- 03:00:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12163m 8583Mi am-6d96779b5d-qbg7l 9742m 8483Mi am-6d96779b5d-zn69c 11436m 8485Mi ds-cts-0 10697m 13516Mi ds-cts-1 10930m 13512Mi ds-cts-2 10449m 13519Mi ds-idrepo-0 7877m 5594Mi ds-idrepo-1 4292m 5523Mi ds-idrepo-2 4168m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 12m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5233m 4191Mi 03:00:16 DEBUG --- stderr --- 03:00:16 DEBUG 03:00:22 INFO 03:00:22 INFO [loop_until]: kubectl --namespace=xlou top node 03:00:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:00:23 INFO [loop_until]: OK (rc = 0) 03:00:23 DEBUG --- stdout --- 03:00:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 4736Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1860Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10223m 34% 10357Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13955m 46% 3098Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12164m 40% 10293Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 4357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12805m 42% 10435Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4348m 27% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10926m 68% 14430Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10917m 68% 14396Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4284m 26% 6695Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10842m 68% 14438Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8217m 51% 6384Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5169m 32% 7938Mi 13% 03:00:23 DEBUG --- stderr --- 03:00:23 DEBUG 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-idrepo-0" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "1752.9333333333334" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "314670" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973663" 03:01:03 ERROR [http_cmd]: ERROR 03:01:03 ERROR --- status code --- 03:01:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:01:03 ERROR --- http response --- 03:01:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:01:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "instance": "10.142.0.107:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "instance": "10.142.0.77:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "668852614144" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0.023" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "1733.1666666666667" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-cts-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "28.099999999999998" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "1737.5" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-cts-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "instance": "10.142.0.107:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "instance": "10.142.0.77:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0.13416666666665691" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "instance": "10.142.0.107:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "instance": "10.142.0.77:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "51326737.166666664" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-cts-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:01:03 DEBUG "instance": "10.142.0.107:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "idm-b85f9d978-b559t" 03:01:03 DEBUG "instance": "10.142.0.77:9100" 03:01:03 DEBUG }, 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG "51323381.5" 03:01:03 DEBUG ] 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "5.2114298416116664" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG { 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG "data": { 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:01:03 DEBUG }, 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG "value": [ 03:01:03 DEBUG { 03:01:03 DEBUG 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG 03:01:03 DEBUG "data": { 03:01:03 DEBUG 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG [...] 03:01:03 DEBUG "result": [ 03:01:03 DEBUG 03:01:03 DEBUG { 03:01:03 DEBUG 03:01:03 DEBUG "metric": { 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:03 DEBUG }, 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG 03:01:03 DEBUG "1737.3" 03:01:03 DEBUG 03:01:03 DEBUG ] 03:01:03 DEBUG [...] 03:01:03 DEBUG } 03:01:03 DEBUG 03:01:03 DEBUG ] 03:01:03 DEBUG 03:01:03 DEBUG } 03:01:03 DEBUG 03:01:03 DEBUG } 03:01:03 DEBUG "pod": "ds-cts-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-idrepo-0" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "ds-idrepo-1" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "ds-idrepo-0" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973663" 03:01:03 INFO [http_cmd]: http status code OK 03:01:03 DEBUG --- status code --- 03:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:03 DEBUG --- http response --- 03:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:01:03 DEBUG ----- output ----- 03:01:03 DEBUG { 03:01:03 DEBUG "status": "success", 03:01:03 DEBUG "data": { 03:01:03 DEBUG "resultType": "vector", 03:01:03 DEBUG "result": [ 03:01:03 DEBUG { 03:01:03 DEBUG "metric": { 03:01:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG [...] 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG 03:01:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:01:03 DEBUG }, 03:01:03 DEBUG "value": [ 03:01:03 DEBUG 1712973663, 03:01:03 DEBUG "0" 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG ] 03:01:03 DEBUG } 03:01:03 DEBUG } 03:01:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973664" 03:01:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973664" 03:01:04 INFO [http_cmd]: http status code OK 03:01:04 DEBUG --- status code --- 03:01:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:04 DEBUG --- http response --- 03:01:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:01:04 DEBUG ----- output ----- 03:01:04 DEBUG { 03:01:04 DEBUG "status": "success", 03:01:04 DEBUG "data": { 03:01:04 DEBUG "resultType": "vector", 03:01:04 DEBUG "result": [ 03:01:04 DEBUG { 03:01:04 DEBUG "metric": { 03:01:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG [...] 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 1712973664, 03:01:04 DEBUG "0" 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG } 03:01:04 INFO [http_cmd]: http status code OK 03:01:04 DEBUG --- status code --- 03:01:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:04 DEBUG --- http response --- 03:01:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:04 DEBUG ----- output ----- 03:01:04 DEBUG { 03:01:04 DEBUG "status": "success", 03:01:04 DEBUG "data": { 03:01:04 DEBUG "resultType": "vector", 03:01:04 DEBUG "result": [ 03:01:04 DEBUG { 03:01:04 DEBUG "metric": { 03:01:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG [...] 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 1712973664, 03:01:04 DEBUG "6238474240" 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG } 03:01:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973664" 03:01:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973664" 03:01:04 INFO [http_cmd]: http status code OK 03:01:04 DEBUG --- status code --- 03:01:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:04 DEBUG --- http response --- 03:01:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:01:04 DEBUG ----- output ----- 03:01:04 DEBUG { 03:01:04 DEBUG "status": "success", 03:01:04 DEBUG "data": { 03:01:04 DEBUG "resultType": "vector", 03:01:04 DEBUG "result": [ 03:01:04 DEBUG { 03:01:04 DEBUG "metric": { 03:01:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG [...] 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 1712973664, 03:01:04 DEBUG "10.412714881673434" 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG } 03:01:04 INFO [http_cmd]: http status code OK 03:01:04 DEBUG --- status code --- 03:01:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:01:04 DEBUG --- http response --- 03:01:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:01:04 DEBUG ----- output ----- 03:01:04 DEBUG { 03:01:04 DEBUG "status": "success", 03:01:04 DEBUG "data": { 03:01:04 DEBUG "resultType": "vector", 03:01:04 DEBUG "result": [ 03:01:04 DEBUG { 03:01:04 DEBUG "metric": { 03:01:04 DEBUG "instance": "10.142.0.107:9100" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG [...] 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG 03:01:04 DEBUG "instance": "10.142.0.77:9100" 03:01:04 DEBUG }, 03:01:04 DEBUG "value": [ 03:01:04 DEBUG 1712973664, 03:01:04 DEBUG "0.04566666666666799" 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG ] 03:01:04 DEBUG } 03:01:04 DEBUG } 03:01:16 INFO 03:01:16 INFO [loop_until]: kubectl --namespace=xlou top pods 03:01:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:01:16 INFO [loop_until]: OK (rc = 0) 03:01:16 DEBUG --- stdout --- 03:01:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12135m 8568Mi am-6d96779b5d-qbg7l 10046m 8480Mi am-6d96779b5d-zn69c 11672m 8480Mi ds-cts-0 10774m 13517Mi ds-cts-1 11009m 13653Mi ds-cts-2 10750m 13520Mi ds-idrepo-0 8231m 5594Mi ds-idrepo-1 4360m 5523Mi ds-idrepo-2 4224m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1120Mi idm-b85f9d978-dz57p 3m 1116Mi lodemon-8dc49bd56-qr24k 7m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5079m 4192Mi 03:01:16 DEBUG --- stderr --- 03:01:16 DEBUG 03:01:23 INFO 03:01:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:01:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:01:23 INFO [loop_until]: OK (rc = 0) 03:01:23 DEBUG --- stdout --- 03:01:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1861Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9918m 33% 10363Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14047m 47% 3102Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12070m 40% 10286Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4340Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12803m 42% 10411Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4415m 27% 6301Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10930m 68% 14418Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11357m 71% 14459Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4245m 26% 6699Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10978m 69% 14439Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8064m 50% 6395Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5380m 33% 7939Mi 13% 03:01:23 DEBUG --- stderr --- 03:01:23 DEBUG 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-idrepo-0" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "1749.1666666666667" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "313687.5" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "instance": "10.142.0.107:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "instance": "10.142.0.77:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "671529833472" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0.023" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 ERROR [http_cmd]: ERROR 03:02:03 ERROR --- status code --- 03:02:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:02:03 ERROR --- http response --- 03:02:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:02:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "1734.6333333333334" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-cts-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0.00092245" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "1730.1333333333332" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-cts-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "29.7" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "3941.5747327903223" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "instance": "10.142.0.107:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "instance": "10.142.0.77:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0.18963333333332266" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "3378.4926281059907" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "instance": "10.142.0.107:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "instance": "10.142.0.77:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "51315442.8" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-cts-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "instance": "10.142.0.107:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "instance": "10.142.0.77:9100" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "51312027.43333334" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-cts-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973723" 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "5.237139207685464" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "1730.1" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-idrepo-0" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973723" 03:02:03 INFO [http_cmd]: http status code OK 03:02:03 DEBUG --- status code --- 03:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:03 DEBUG --- http response --- 03:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:03 DEBUG ----- output ----- 03:02:03 DEBUG { 03:02:03 DEBUG "status": "success", 03:02:03 DEBUG "data": { 03:02:03 DEBUG "resultType": "vector", 03:02:03 DEBUG "result": [ 03:02:03 DEBUG { 03:02:03 DEBUG "metric": { 03:02:03 DEBUG "pod": "ds-idrepo-1" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG [...] 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG 03:02:03 DEBUG "pod": "ds-idrepo-0" 03:02:03 DEBUG }, 03:02:03 DEBUG "value": [ 03:02:03 DEBUG 1712973723, 03:02:03 DEBUG "0" 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG ] 03:02:03 DEBUG } 03:02:03 DEBUG } 03:02:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973724" 03:02:04 INFO [http_cmd]: http status code OK 03:02:04 DEBUG --- status code --- 03:02:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:04 DEBUG --- http response --- 03:02:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:02:04 DEBUG ----- output ----- 03:02:04 DEBUG { 03:02:04 DEBUG "status": "success", 03:02:04 DEBUG "data": { 03:02:04 DEBUG "resultType": "vector", 03:02:04 DEBUG "result": [ 03:02:04 DEBUG { 03:02:04 DEBUG "metric": { 03:02:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG [...] 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG "pod": "ds-idrepo-1" 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 1712973724, 03:02:04 DEBUG "0" 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973724" 03:02:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973724" 03:02:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973724" 03:02:04 INFO [http_cmd]: http status code OK 03:02:04 INFO [http_cmd]: http status code OK 03:02:04 DEBUG --- status code --- 03:02:04 DEBUG --- status code --- 03:02:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:04 DEBUG --- http response --- 03:02:04 DEBUG --- http response --- 03:02:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:02:04 DEBUG ----- output ----- 03:02:04 DEBUG ----- output ----- 03:02:04 DEBUG { 03:02:04 DEBUG { 03:02:04 DEBUG "status": "success", 03:02:04 DEBUG "status": "success", 03:02:04 DEBUG "data": { 03:02:04 DEBUG "data": { 03:02:04 DEBUG "resultType": "vector", 03:02:04 DEBUG "resultType": "vector", 03:02:04 DEBUG "result": [ 03:02:04 DEBUG "result": [ 03:02:04 DEBUG { 03:02:04 DEBUG { 03:02:04 DEBUG "metric": { 03:02:04 DEBUG "metric": { 03:02:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:02:04 DEBUG "instance": "10.142.0.107:9100" 03:02:04 DEBUG }, 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG [...] 03:02:04 DEBUG [...] 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:02:04 DEBUG "instance": "10.142.0.77:9100" 03:02:04 DEBUG }, 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 1712973724, 03:02:04 DEBUG 1712973724, 03:02:04 DEBUG "6236676096" 03:02:04 DEBUG "0.06000000000000085" 03:02:04 DEBUG ] 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:04 DEBUG ] 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:04 INFO [http_cmd]: http status code OK 03:02:04 DEBUG --- status code --- 03:02:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:04 DEBUG --- http response --- 03:02:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:02:04 DEBUG ----- output ----- 03:02:04 DEBUG { 03:02:04 DEBUG "status": "success", 03:02:04 DEBUG "data": { 03:02:04 DEBUG "resultType": "vector", 03:02:04 DEBUG "result": [ 03:02:04 DEBUG { 03:02:04 DEBUG "metric": { 03:02:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG [...] 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 1712973724, 03:02:04 DEBUG "0" 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973724" 03:02:04 INFO [http_cmd]: http status code OK 03:02:04 DEBUG --- status code --- 03:02:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:02:04 DEBUG --- http response --- 03:02:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:02:04 DEBUG ----- output ----- 03:02:04 DEBUG { 03:02:04 DEBUG "status": "success", 03:02:04 DEBUG "data": { 03:02:04 DEBUG "resultType": "vector", 03:02:04 DEBUG "result": [ 03:02:04 DEBUG { 03:02:04 DEBUG "metric": { 03:02:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG [...] 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG 03:02:04 DEBUG "pod": "ds-idrepo-1" 03:02:04 DEBUG }, 03:02:04 DEBUG "value": [ 03:02:04 DEBUG 1712973724, 03:02:04 DEBUG "4.317980762071819" 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG ] 03:02:04 DEBUG } 03:02:04 DEBUG } 03:02:16 INFO 03:02:16 INFO [loop_until]: kubectl --namespace=xlou top pods 03:02:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:02:16 INFO [loop_until]: OK (rc = 0) 03:02:16 DEBUG --- stdout --- 03:02:16 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12513m 8568Mi am-6d96779b5d-qbg7l 9799m 8480Mi am-6d96779b5d-zn69c 11735m 8484Mi ds-cts-0 10876m 13517Mi ds-cts-1 10688m 13513Mi ds-cts-2 10581m 13519Mi ds-idrepo-0 8108m 5594Mi ds-idrepo-1 4307m 5523Mi ds-idrepo-2 4141m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 6m 1121Mi lodemon-8dc49bd56-qr24k 5m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5300m 4192Mi 03:02:16 DEBUG --- stderr --- 03:02:16 DEBUG 03:02:23 INFO 03:02:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:02:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:02:23 INFO [loop_until]: OK (rc = 0) 03:02:23 DEBUG --- stdout --- 03:02:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4741Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1869Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10369m 34% 10362Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14121m 47% 3112Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11997m 40% 10291Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 4338Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12792m 42% 10417Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4271m 26% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10798m 67% 14421Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10941m 68% 14380Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4224m 26% 6699Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10900m 68% 14435Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8252m 51% 6398Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5452m 34% 7939Mi 13% 03:02:23 DEBUG --- stderr --- 03:02:23 DEBUG 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-idrepo-0" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "1740.0666666666666" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:03 DEBUG "instance": "10.142.0.107:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "instance": "10.142.0.77:9100" 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG }, 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "674302868480" 03:03:03 DEBUG "313234.5" 03:03:03 DEBUG ] 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 ERROR [http_cmd]: ERROR 03:03:03 ERROR --- status code --- 03:03:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:03:03 ERROR --- http response --- 03:03:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:03:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0.023" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "1749.8" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-cts-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "28.633333333333333" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-cts-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0.0002" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "1749.4333333333334" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "instance": "10.142.0.107:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "instance": "10.142.0.77:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0.18880000000000527" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:03:03 DEBUG "instance": "10.142.0.107:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG "instance": "10.142.0.77:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG "51528519.099999994" 03:03:03 DEBUG ] 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "instance": "10.142.0.107:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "instance": "10.142.0.77:9100" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "51532280.3" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-cts-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-cts-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973783" 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "5.284435303517734" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "1749.1333333333332" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-idrepo-0" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973783" 03:03:03 INFO [http_cmd]: http status code OK 03:03:03 DEBUG --- status code --- 03:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:03 DEBUG --- http response --- 03:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:03 DEBUG ----- output ----- 03:03:03 DEBUG { 03:03:03 DEBUG "status": "success", 03:03:03 DEBUG "data": { 03:03:03 DEBUG "resultType": "vector", 03:03:03 DEBUG "result": [ 03:03:03 DEBUG { 03:03:03 DEBUG "metric": { 03:03:03 DEBUG "pod": "ds-idrepo-1" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG [...] 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG 03:03:03 DEBUG "pod": "ds-idrepo-0" 03:03:03 DEBUG }, 03:03:03 DEBUG "value": [ 03:03:03 DEBUG 1712973783, 03:03:03 DEBUG "0" 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG ] 03:03:03 DEBUG } 03:03:03 DEBUG } 03:03:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973784" 03:03:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973784" 03:03:04 INFO [http_cmd]: http status code OK 03:03:04 DEBUG --- status code --- 03:03:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:04 DEBUG --- http response --- 03:03:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:04 DEBUG ----- output ----- 03:03:04 DEBUG { 03:03:04 DEBUG "status": "success", 03:03:04 DEBUG "data": { 03:03:04 DEBUG "resultType": "vector", 03:03:04 DEBUG "result": [ 03:03:04 DEBUG { 03:03:04 DEBUG "metric": { 03:03:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG [...] 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 1712973784, 03:03:04 DEBUG "0" 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG } 03:03:04 INFO [http_cmd]: http status code OK 03:03:04 DEBUG --- status code --- 03:03:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:04 DEBUG --- http response --- 03:03:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:04 DEBUG ----- output ----- 03:03:04 DEBUG { 03:03:04 DEBUG "status": "success", 03:03:04 DEBUG "data": { 03:03:04 DEBUG "resultType": "vector", 03:03:04 DEBUG "result": [ 03:03:04 DEBUG { 03:03:04 DEBUG "metric": { 03:03:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG [...] 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 1712973784, 03:03:04 DEBUG "6237343744" 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG } 03:03:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973784" 03:03:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973784" 03:03:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973784" 03:03:04 INFO [http_cmd]: http status code OK 03:03:04 DEBUG --- status code --- 03:03:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:04 DEBUG --- http response --- 03:03:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:03:04 DEBUG ----- output ----- 03:03:04 DEBUG { 03:03:04 DEBUG "status": "success", 03:03:04 DEBUG "data": { 03:03:04 DEBUG "resultType": "vector", 03:03:04 DEBUG "result": [ 03:03:04 DEBUG { 03:03:04 DEBUG "metric": { 03:03:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG [...] 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 1712973784, 03:03:04 DEBUG "0" 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG } 03:03:04 INFO [http_cmd]: http status code OK 03:03:04 DEBUG --- status code --- 03:03:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:04 DEBUG --- http response --- 03:03:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:03:04 DEBUG ----- output ----- 03:03:04 DEBUG { 03:03:04 DEBUG "status": "success", 03:03:04 DEBUG "data": { 03:03:04 DEBUG "resultType": "vector", 03:03:04 DEBUG "result": [ 03:03:04 DEBUG { 03:03:04 DEBUG "metric": { 03:03:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG [...] 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 1712973784, 03:03:04 DEBUG "0" 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG } 03:03:04 INFO [http_cmd]: http status code OK 03:03:04 DEBUG --- status code --- 03:03:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:03:04 DEBUG --- http response --- 03:03:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:03:04 DEBUG ----- output ----- 03:03:04 DEBUG { 03:03:04 DEBUG "status": "success", 03:03:04 DEBUG "data": { 03:03:04 DEBUG "resultType": "vector", 03:03:04 DEBUG "result": [ 03:03:04 DEBUG { 03:03:04 DEBUG "metric": { 03:03:04 DEBUG "instance": "10.142.0.107:9100" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG [...] 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG 03:03:04 DEBUG "instance": "10.142.0.77:9100" 03:03:04 DEBUG }, 03:03:04 DEBUG "value": [ 03:03:04 DEBUG 1712973784, 03:03:04 DEBUG "0.061000000000001595" 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG ] 03:03:04 DEBUG } 03:03:04 DEBUG } 03:03:17 INFO 03:03:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:03:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:03:17 INFO [loop_until]: OK (rc = 0) 03:03:17 DEBUG --- stdout --- 03:03:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12336m 8569Mi am-6d96779b5d-qbg7l 9711m 8482Mi am-6d96779b5d-zn69c 12163m 8483Mi ds-cts-0 10652m 13516Mi ds-cts-1 11085m 13619Mi ds-cts-2 10533m 13519Mi ds-idrepo-0 7857m 5594Mi ds-idrepo-1 4449m 5523Mi ds-idrepo-2 4067m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5217m 4193Mi 03:03:17 DEBUG --- stderr --- 03:03:17 DEBUG 03:03:23 INFO 03:03:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:03:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:03:23 INFO [loop_until]: OK (rc = 0) 03:03:23 DEBUG --- stdout --- 03:03:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4737Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10224m 34% 10359Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14063m 47% 3109Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12246m 41% 10287Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4348Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12802m 42% 10419Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4423m 27% 6305Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10501m 66% 14431Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10799m 67% 14546Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4221m 26% 6698Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10497m 66% 14442Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8284m 52% 6386Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5381m 33% 7939Mi 13% 03:03:23 DEBUG --- stderr --- 03:03:23 DEBUG 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-idrepo-0" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "1699.9666666666667" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "313845.5" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:03 DEBUG }, 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "1736.0333333333333" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "instance": "10.142.0.107:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "instance": "10.142.0.77:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "677082418176" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 ERROR [http_cmd]: ERROR 03:04:03 ERROR --- status code --- 03:04:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:04:03 ERROR --- http response --- 03:04:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:04:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0.023" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-cts-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "28.333333333333332" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "1727.5666666666666" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-cts-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "instance": "10.142.0.107:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "instance": "10.142.0.77:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0.16596666666666002" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "idm-b85f9d978-b559t" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "instance": "10.142.0.107:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "instance": "10.142.0.77:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "50636519.9" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "instance": "10.142.0.107:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "instance": "10.142.0.77:9100" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "50640112.599999994" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-cts-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-cts-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "1727.3666666666666" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "5.231216299090452" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-idrepo-0" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973843" 03:04:03 INFO [http_cmd]: http status code OK 03:04:03 DEBUG --- status code --- 03:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:03 DEBUG --- http response --- 03:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:03 DEBUG ----- output ----- 03:04:03 DEBUG { 03:04:03 DEBUG "status": "success", 03:04:03 DEBUG "data": { 03:04:03 DEBUG "resultType": "vector", 03:04:03 DEBUG "result": [ 03:04:03 DEBUG { 03:04:03 DEBUG "metric": { 03:04:03 DEBUG "pod": "ds-idrepo-1" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG [...] 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG 03:04:03 DEBUG "pod": "ds-idrepo-0" 03:04:03 DEBUG }, 03:04:03 DEBUG "value": [ 03:04:03 DEBUG 1712973843, 03:04:03 DEBUG "0" 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG ] 03:04:03 DEBUG } 03:04:03 DEBUG } 03:04:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973844" 03:04:04 INFO [http_cmd]: http status code OK 03:04:04 DEBUG --- status code --- 03:04:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:04 DEBUG --- http response --- 03:04:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:04:04 DEBUG ----- output ----- 03:04:04 DEBUG { 03:04:04 DEBUG "status": "success", 03:04:04 DEBUG "data": { 03:04:04 DEBUG "resultType": "vector", 03:04:04 DEBUG "result": [ 03:04:04 DEBUG { 03:04:04 DEBUG "metric": { 03:04:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG [...] 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG "pod": "ds-cts-1" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 1712973844, 03:04:04 DEBUG "12939445.733788397" 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG } 03:04:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973844" 03:04:04 INFO [http_cmd]: http status code OK 03:04:04 DEBUG --- status code --- 03:04:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:04 DEBUG --- http response --- 03:04:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:04 DEBUG ----- output ----- 03:04:04 DEBUG { 03:04:04 DEBUG "status": "success", 03:04:04 DEBUG "data": { 03:04:04 DEBUG "resultType": "vector", 03:04:04 DEBUG "result": [ 03:04:04 DEBUG { 03:04:04 DEBUG "metric": { 03:04:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG [...] 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 1712973844, 03:04:04 DEBUG "6237577216" 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG } 03:04:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973844" 03:04:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973844" 03:04:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973844" 03:04:04 INFO [http_cmd]: http status code OK 03:04:04 DEBUG --- status code --- 03:04:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:04 DEBUG --- http response --- 03:04:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:04:04 DEBUG ----- output ----- 03:04:04 DEBUG { 03:04:04 DEBUG "status": "success", 03:04:04 DEBUG "data": { 03:04:04 DEBUG "resultType": "vector", 03:04:04 DEBUG "result": [ 03:04:04 DEBUG { 03:04:04 DEBUG "metric": { 03:04:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG [...] 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 1712973844, 03:04:04 DEBUG "0" 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG } 03:04:04 INFO [http_cmd]: http status code OK 03:04:04 DEBUG --- status code --- 03:04:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:04 DEBUG --- http response --- 03:04:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:04:04 DEBUG ----- output ----- 03:04:04 DEBUG { 03:04:04 DEBUG "status": "success", 03:04:04 DEBUG "data": { 03:04:04 DEBUG "resultType": "vector", 03:04:04 DEBUG "result": [ 03:04:04 DEBUG { 03:04:04 DEBUG "metric": { 03:04:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG [...] 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG "pod": "ds-cts-1" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 1712973844, 03:04:04 DEBUG "10.942178944911207" 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG } 03:04:04 INFO [http_cmd]: http status code OK 03:04:04 DEBUG --- status code --- 03:04:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:04:04 DEBUG --- http response --- 03:04:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:04:04 DEBUG ----- output ----- 03:04:04 DEBUG { 03:04:04 DEBUG "status": "success", 03:04:04 DEBUG "data": { 03:04:04 DEBUG "resultType": "vector", 03:04:04 DEBUG "result": [ 03:04:04 DEBUG { 03:04:04 DEBUG "metric": { 03:04:04 DEBUG "instance": "10.142.0.107:9100" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG [...] 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG 03:04:04 DEBUG "instance": "10.142.0.77:9100" 03:04:04 DEBUG }, 03:04:04 DEBUG "value": [ 03:04:04 DEBUG 1712973844, 03:04:04 DEBUG "0.043666666666665556" 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG ] 03:04:04 DEBUG } 03:04:04 DEBUG } 03:04:17 INFO 03:04:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:04:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:04:17 INFO [loop_until]: OK (rc = 0) 03:04:17 DEBUG --- stdout --- 03:04:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12190m 8573Mi am-6d96779b5d-qbg7l 10008m 8495Mi am-6d96779b5d-zn69c 11721m 8483Mi ds-cts-0 10743m 13518Mi ds-cts-1 11139m 13590Mi ds-cts-2 10897m 13519Mi ds-idrepo-0 8209m 5594Mi ds-idrepo-1 4353m 5523Mi ds-idrepo-2 4189m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 2m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5250m 4192Mi 03:04:17 DEBUG --- stderr --- 03:04:17 DEBUG 03:04:23 INFO 03:04:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:04:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:04:23 INFO [loop_until]: OK (rc = 0) 03:04:23 DEBUG --- stdout --- 03:04:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4741Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10263m 34% 10371Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14042m 47% 3090Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12131m 40% 10290Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4339Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12926m 43% 10425Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4356m 27% 6305Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10794m 67% 14426Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11218m 70% 14444Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4230m 26% 6699Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10925m 68% 14437Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8247m 51% 6382Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5208m 32% 7939Mi 13% 03:04:23 DEBUG --- stderr --- 03:04:23 DEBUG 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-idrepo-0" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "1736.4333333333334" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "313731" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "1733.4" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "instance": "10.142.0.107:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "instance": "10.142.0.77:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "679885837312" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 ERROR [http_cmd]: ERROR 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 ERROR --- status code --- 03:05:03 DEBUG --- status code --- 03:05:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 ERROR --- http response --- 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:05:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0.023" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-cts-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "28.333333333333332" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "1734.5333333333333" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-cts-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0.000048549999999999994" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "instance": "10.142.0.107:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "instance": "10.142.0.77:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0.17539999999999384" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "instance": "10.142.0.107:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "instance": "10.142.0.77:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "51401976.333333336" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "6558047.047211713" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:05:03 DEBUG "instance": "10.142.0.107:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG "instance": "10.142.0.77:9100" 03:05:03 DEBUG }, 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "295835907.7479849" 03:05:03 DEBUG "51398603.099999994" 03:05:03 DEBUG ] 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-cts-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-cts-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "5.245376577076223" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "1735.0666666666666" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-idrepo-0" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973903" 03:05:03 INFO [http_cmd]: http status code OK 03:05:03 DEBUG --- status code --- 03:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:03 DEBUG --- http response --- 03:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:03 DEBUG ----- output ----- 03:05:03 DEBUG { 03:05:03 DEBUG "status": "success", 03:05:03 DEBUG "data": { 03:05:03 DEBUG "resultType": "vector", 03:05:03 DEBUG "result": [ 03:05:03 DEBUG { 03:05:03 DEBUG "metric": { 03:05:03 DEBUG "pod": "ds-idrepo-1" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG [...] 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG 03:05:03 DEBUG "pod": "ds-idrepo-0" 03:05:03 DEBUG }, 03:05:03 DEBUG "value": [ 03:05:03 DEBUG 1712973903, 03:05:03 DEBUG "0" 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG ] 03:05:03 DEBUG } 03:05:03 DEBUG } 03:05:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973904" 03:05:04 INFO [http_cmd]: http status code OK 03:05:04 DEBUG --- status code --- 03:05:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:04 DEBUG --- http response --- 03:05:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:05:04 DEBUG ----- output ----- 03:05:04 DEBUG { 03:05:04 DEBUG "status": "success", 03:05:04 DEBUG "data": { 03:05:04 DEBUG "resultType": "vector", 03:05:04 DEBUG "result": [ 03:05:04 DEBUG { 03:05:04 DEBUG "metric": { 03:05:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG [...] 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 1712973904, 03:05:04 DEBUG "0" 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG } 03:05:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973904" 03:05:04 INFO [http_cmd]: http status code OK 03:05:04 DEBUG --- status code --- 03:05:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:04 DEBUG --- http response --- 03:05:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:04 DEBUG ----- output ----- 03:05:04 DEBUG { 03:05:04 DEBUG "status": "success", 03:05:04 DEBUG "data": { 03:05:04 DEBUG "resultType": "vector", 03:05:04 DEBUG "result": [ 03:05:04 DEBUG { 03:05:04 DEBUG "metric": { 03:05:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG [...] 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 1712973904, 03:05:04 DEBUG "6238904320" 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG } 03:05:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973904" 03:05:04 INFO [http_cmd]: http status code OK 03:05:04 DEBUG --- status code --- 03:05:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:04 DEBUG --- http response --- 03:05:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:05:04 DEBUG ----- output ----- 03:05:04 DEBUG { 03:05:04 DEBUG "status": "success", 03:05:04 DEBUG "data": { 03:05:04 DEBUG "resultType": "vector", 03:05:04 DEBUG "result": [ 03:05:04 DEBUG { 03:05:04 DEBUG "metric": { 03:05:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG [...] 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 1712973904, 03:05:04 DEBUG "0" 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG } 03:05:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973904" 03:05:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973904" 03:05:04 INFO [http_cmd]: http status code OK 03:05:04 DEBUG --- status code --- 03:05:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:04 DEBUG --- http response --- 03:05:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:05:04 DEBUG ----- output ----- 03:05:04 DEBUG { 03:05:04 DEBUG "status": "success", 03:05:04 DEBUG "data": { 03:05:04 DEBUG "resultType": "vector", 03:05:04 DEBUG "result": [ 03:05:04 DEBUG { 03:05:04 DEBUG "metric": { 03:05:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG [...] 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 1712973904, 03:05:04 DEBUG "12.06040407961867" 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG } 03:05:04 INFO [http_cmd]: http status code OK 03:05:04 DEBUG --- status code --- 03:05:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:05:04 DEBUG --- http response --- 03:05:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:05:04 DEBUG ----- output ----- 03:05:04 DEBUG { 03:05:04 DEBUG "status": "success", 03:05:04 DEBUG "data": { 03:05:04 DEBUG "resultType": "vector", 03:05:04 DEBUG "result": [ 03:05:04 DEBUG { 03:05:04 DEBUG "metric": { 03:05:04 DEBUG "instance": "10.142.0.107:9100" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG [...] 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG 03:05:04 DEBUG "instance": "10.142.0.77:9100" 03:05:04 DEBUG }, 03:05:04 DEBUG "value": [ 03:05:04 DEBUG 1712973904, 03:05:04 DEBUG "0.05199999999999936" 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG ] 03:05:04 DEBUG } 03:05:04 DEBUG } 03:05:17 INFO 03:05:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:05:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:05:17 INFO [loop_until]: OK (rc = 0) 03:05:17 DEBUG --- stdout --- 03:05:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12187m 8567Mi am-6d96779b5d-qbg7l 9731m 8485Mi am-6d96779b5d-zn69c 11515m 8481Mi ds-cts-0 10554m 13518Mi ds-cts-1 11376m 13513Mi ds-cts-2 10835m 13518Mi ds-idrepo-0 8309m 5594Mi ds-idrepo-1 4382m 5523Mi ds-idrepo-2 4036m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5231m 4193Mi 03:05:17 DEBUG --- stderr --- 03:05:17 DEBUG 03:05:23 INFO 03:05:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:05:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:05:23 INFO [loop_until]: OK (rc = 0) 03:05:23 DEBUG --- stdout --- 03:05:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4740Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10222m 34% 10363Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14059m 47% 3091Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12051m 40% 10287Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 133m 0% 4355Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12768m 42% 10426Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4386m 27% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10913m 68% 14424Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11312m 71% 14395Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4107m 25% 6702Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10892m 68% 14428Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8251m 51% 6386Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5372m 33% 7935Mi 13% 03:05:23 DEBUG --- stderr --- 03:05:23 DEBUG 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-idrepo-0" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "1749.8" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "313566" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "instance": "10.142.0.107:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "instance": "10.142.0.77:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "682481218560" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "1751.5" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 ERROR [http_cmd]: ERROR 03:06:03 ERROR --- status code --- 03:06:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:06:03 ERROR --- http response --- 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:06:03 DEBUG --- status code --- 03:06:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0.023" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-cts-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "28.966666666666665" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "1750.8999999999999" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-cts-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0.000655425" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "instance": "10.142.0.107:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "instance": "10.142.0.77:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0.17739999999999345" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "instance": "10.142.0.107:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "instance": "10.142.0.77:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "51255226.8" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "instance": "10.142.0.107:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "instance": "10.142.0.77:9100" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "51251334.13333333" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "42763.15408462395" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-cts-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-cts-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "5.245514793299498" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "1751.0333333333333" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-idrepo-0" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973963" 03:06:03 INFO [http_cmd]: http status code OK 03:06:03 DEBUG --- status code --- 03:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:03 DEBUG --- http response --- 03:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:03 DEBUG ----- output ----- 03:06:03 DEBUG { 03:06:03 DEBUG "status": "success", 03:06:03 DEBUG "data": { 03:06:03 DEBUG "resultType": "vector", 03:06:03 DEBUG "result": [ 03:06:03 DEBUG { 03:06:03 DEBUG "metric": { 03:06:03 DEBUG "pod": "ds-idrepo-1" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG [...] 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG 03:06:03 DEBUG "pod": "ds-idrepo-0" 03:06:03 DEBUG }, 03:06:03 DEBUG "value": [ 03:06:03 DEBUG 1712973963, 03:06:03 DEBUG "0" 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG ] 03:06:03 DEBUG } 03:06:03 DEBUG } 03:06:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973964" 03:06:04 INFO [http_cmd]: http status code OK 03:06:04 DEBUG --- status code --- 03:06:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:04 DEBUG --- http response --- 03:06:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:06:04 DEBUG ----- output ----- 03:06:04 DEBUG { 03:06:04 DEBUG "status": "success", 03:06:04 DEBUG "data": { 03:06:04 DEBUG "resultType": "vector", 03:06:04 DEBUG "result": [ 03:06:04 DEBUG { 03:06:04 DEBUG "metric": { 03:06:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG [...] 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 1712973964, 03:06:04 DEBUG "0" 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG } 03:06:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712973964" 03:06:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973964" 03:06:04 INFO [http_cmd]: http status code OK 03:06:04 DEBUG --- status code --- 03:06:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:04 DEBUG --- http response --- 03:06:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:04 DEBUG ----- output ----- 03:06:04 DEBUG { 03:06:04 DEBUG "status": "success", 03:06:04 DEBUG "data": { 03:06:04 DEBUG "resultType": "vector", 03:06:04 DEBUG "result": [ 03:06:04 DEBUG { 03:06:04 DEBUG "metric": { 03:06:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG [...] 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 1712973964, 03:06:04 DEBUG "6241095680" 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG } 03:06:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712973964" 03:06:04 INFO [http_cmd]: http status code OK 03:06:04 DEBUG --- status code --- 03:06:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:04 DEBUG --- http response --- 03:06:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:06:04 DEBUG ----- output ----- 03:06:04 DEBUG { 03:06:04 DEBUG "status": "success", 03:06:04 DEBUG "data": { 03:06:04 DEBUG "resultType": "vector", 03:06:04 DEBUG "result": [ 03:06:04 DEBUG { 03:06:04 DEBUG "metric": { 03:06:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG [...] 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 1712973964, 03:06:04 DEBUG "12.072536720044576" 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG } 03:06:04 INFO [http_cmd]: http status code OK 03:06:04 DEBUG --- status code --- 03:06:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:04 DEBUG --- http response --- 03:06:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:06:04 DEBUG ----- output ----- 03:06:04 DEBUG { 03:06:04 DEBUG "status": "success", 03:06:04 DEBUG "data": { 03:06:04 DEBUG "resultType": "vector", 03:06:04 DEBUG "result": [ 03:06:04 DEBUG { 03:06:04 DEBUG "metric": { 03:06:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG [...] 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 1712973964, 03:06:04 DEBUG "0" 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG } 03:06:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712973964" 03:06:04 INFO [http_cmd]: http status code OK 03:06:04 DEBUG --- status code --- 03:06:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:06:04 DEBUG --- http response --- 03:06:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:06:04 DEBUG ----- output ----- 03:06:04 DEBUG { 03:06:04 DEBUG "status": "success", 03:06:04 DEBUG "data": { 03:06:04 DEBUG "resultType": "vector", 03:06:04 DEBUG "result": [ 03:06:04 DEBUG { 03:06:04 DEBUG "metric": { 03:06:04 DEBUG "instance": "10.142.0.107:9100" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG [...] 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG 03:06:04 DEBUG "instance": "10.142.0.77:9100" 03:06:04 DEBUG }, 03:06:04 DEBUG "value": [ 03:06:04 DEBUG 1712973964, 03:06:04 DEBUG "0.05666666666666581" 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG ] 03:06:04 DEBUG } 03:06:04 DEBUG } 03:06:17 INFO 03:06:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:06:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:06:17 INFO [loop_until]: OK (rc = 0) 03:06:17 DEBUG --- stdout --- 03:06:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12114m 8567Mi am-6d96779b5d-qbg7l 9740m 8480Mi am-6d96779b5d-zn69c 12014m 8480Mi ds-cts-0 10583m 13621Mi ds-cts-1 11226m 13512Mi ds-cts-2 11045m 13520Mi ds-idrepo-0 8195m 5594Mi ds-idrepo-1 4339m 5523Mi ds-idrepo-2 4127m 5861Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5248m 4193Mi 03:06:17 DEBUG --- stderr --- 03:06:17 DEBUG 03:06:23 INFO 03:06:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:06:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:06:23 INFO [loop_until]: OK (rc = 0) 03:06:23 DEBUG --- stdout --- 03:06:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 4753Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1869Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10263m 34% 10360Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13998m 46% 3095Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11646m 39% 10291Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4327Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12762m 42% 10411Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4396m 27% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10639m 66% 14423Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11325m 71% 14401Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4285m 26% 6702Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10754m 67% 14489Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8274m 52% 6384Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5345m 33% 7936Mi 13% 03:06:23 DEBUG --- stderr --- 03:06:23 DEBUG 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-idrepo-0" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "0" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "1738.0666666666666" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "313622.5" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "instance": "10.142.0.107:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "instance": "10.142.0.77:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "684679087104" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "1755.6333333333334" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 ERROR [http_cmd]: ERROR 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 ERROR --- status code --- 03:07:03 DEBUG --- status code --- 03:07:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 ERROR --- http response --- 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:07:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "0.023" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-cts-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "27.5" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "1755.3" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-cts-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "0.0013760666666666668" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "instance": "10.142.0.107:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "instance": "10.142.0.77:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "0.17949999999999305" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "instance": "10.142.0.107:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "instance": "10.142.0.77:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "51392527.86666667" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "2939.169575002446" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "instance": "10.142.0.107:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "instance": "10.142.0.77:9100" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "51388922.7" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:07:03 DEBUG "result": [ 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG "data": { 03:07:03 DEBUG }, 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "value": [ 03:07:03 DEBUG "result": [ 03:07:03 DEBUG 03:07:03 DEBUG { 03:07:03 DEBUG 03:07:03 DEBUG "metric": { 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG [...] 03:07:03 DEBUG }, 03:07:03 DEBUG 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-cts-1" 03:07:03 DEBUG 03:07:03 DEBUG }, 03:07:03 DEBUG [...] 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG 03:07:03 DEBUG "0" 03:07:03 DEBUG 03:07:03 DEBUG ] 03:07:03 DEBUG "pod": "ds-cts-1" 03:07:03 DEBUG } 03:07:03 DEBUG }, 03:07:03 DEBUG ] 03:07:03 DEBUG "value": [ 03:07:03 DEBUG } 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG } 03:07:03 DEBUG "0" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "9485.501810235168" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "5.23352956684937" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-idrepo-0" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "0" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "1755.2666666666667" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974023" 03:07:03 INFO [http_cmd]: http status code OK 03:07:03 DEBUG --- status code --- 03:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:03 DEBUG --- http response --- 03:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:03 DEBUG ----- output ----- 03:07:03 DEBUG { 03:07:03 DEBUG "status": "success", 03:07:03 DEBUG "data": { 03:07:03 DEBUG "resultType": "vector", 03:07:03 DEBUG "result": [ 03:07:03 DEBUG { 03:07:03 DEBUG "metric": { 03:07:03 DEBUG "pod": "ds-idrepo-1" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG [...] 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG 03:07:03 DEBUG "pod": "ds-idrepo-0" 03:07:03 DEBUG }, 03:07:03 DEBUG "value": [ 03:07:03 DEBUG 1712974023, 03:07:03 DEBUG "0" 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG ] 03:07:03 DEBUG } 03:07:03 DEBUG } 03:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974024" 03:07:04 INFO [http_cmd]: http status code OK 03:07:04 DEBUG --- status code --- 03:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:04 DEBUG --- http response --- 03:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:04 DEBUG ----- output ----- 03:07:04 DEBUG { 03:07:04 DEBUG "status": "success", 03:07:04 DEBUG "data": { 03:07:04 DEBUG "resultType": "vector", 03:07:04 DEBUG "result": [ 03:07:04 DEBUG { 03:07:04 DEBUG "metric": { 03:07:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG [...] 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG "pod": "ds-cts-2" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 1712974024, 03:07:04 DEBUG "21126561.61708185" 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG } 03:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974024" 03:07:04 INFO [http_cmd]: http status code OK 03:07:04 DEBUG --- status code --- 03:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:04 DEBUG --- http response --- 03:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:04 DEBUG ----- output ----- 03:07:04 DEBUG { 03:07:04 DEBUG "status": "success", 03:07:04 DEBUG "data": { 03:07:04 DEBUG "resultType": "vector", 03:07:04 DEBUG "result": [ 03:07:04 DEBUG { 03:07:04 DEBUG "metric": { 03:07:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG [...] 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 1712974024, 03:07:04 DEBUG "6241153024" 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG } 03:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974024" 03:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974024" 03:07:04 INFO [http_cmd]: http status code OK 03:07:04 DEBUG --- status code --- 03:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:04 DEBUG --- http response --- 03:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:07:04 DEBUG ----- output ----- 03:07:04 DEBUG { 03:07:04 DEBUG "status": "success", 03:07:04 DEBUG "data": { 03:07:04 DEBUG "resultType": "vector", 03:07:04 DEBUG "result": [ 03:07:04 DEBUG { 03:07:04 DEBUG "metric": { 03:07:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG [...] 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG "pod": "ds-idrepo-1" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 1712974024, 03:07:04 DEBUG "0" 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG } 03:07:04 INFO [http_cmd]: http status code OK 03:07:04 DEBUG --- status code --- 03:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:04 DEBUG --- http response --- 03:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:07:04 DEBUG ----- output ----- 03:07:04 DEBUG { 03:07:04 DEBUG "status": "success", 03:07:04 DEBUG "data": { 03:07:04 DEBUG "resultType": "vector", 03:07:04 DEBUG "result": [ 03:07:04 DEBUG { 03:07:04 DEBUG "metric": { 03:07:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG [...] 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 1712974024, 03:07:04 DEBUG "0" 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG } 03:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974024" 03:07:04 INFO [http_cmd]: http status code OK 03:07:04 DEBUG --- status code --- 03:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:07:04 DEBUG --- http response --- 03:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:07:04 DEBUG ----- output ----- 03:07:04 DEBUG { 03:07:04 DEBUG "status": "success", 03:07:04 DEBUG "data": { 03:07:04 DEBUG "resultType": "vector", 03:07:04 DEBUG "result": [ 03:07:04 DEBUG { 03:07:04 DEBUG "metric": { 03:07:04 DEBUG "instance": "10.142.0.107:9100" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG [...] 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG 03:07:04 DEBUG "instance": "10.142.0.77:9100" 03:07:04 DEBUG }, 03:07:04 DEBUG "value": [ 03:07:04 DEBUG 1712974024, 03:07:04 DEBUG "0.05166666666666752" 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG ] 03:07:04 DEBUG } 03:07:04 DEBUG } 03:07:17 INFO 03:07:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:07:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:07:17 INFO [loop_until]: OK (rc = 0) 03:07:17 DEBUG --- stdout --- 03:07:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11821m 8572Mi am-6d96779b5d-qbg7l 9969m 8479Mi am-6d96779b5d-zn69c 11849m 8488Mi ds-cts-0 10425m 13516Mi ds-cts-1 10912m 13512Mi ds-cts-2 10653m 13519Mi ds-idrepo-0 8169m 5594Mi ds-idrepo-1 4299m 5523Mi ds-idrepo-2 4084m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 2m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5360m 4193Mi 03:07:17 DEBUG --- stderr --- 03:07:17 DEBUG 03:07:23 INFO 03:07:23 INFO [loop_until]: kubectl --namespace=xlou top node 03:07:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:07:23 INFO [loop_until]: OK (rc = 0) 03:07:23 DEBUG --- stdout --- 03:07:23 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4743Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1869Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10134m 33% 10358Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14019m 46% 3089Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12147m 40% 10295Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4334Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12269m 41% 10419Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4402m 27% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10540m 66% 14429Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10579m 66% 14393Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4266m 26% 6704Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10553m 66% 14447Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8257m 51% 6384Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5440m 34% 7930Mi 13% 03:07:23 DEBUG --- stderr --- 03:07:23 DEBUG 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "ds-idrepo-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "ds-idrepo-0" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "0" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "1736.9333333333334" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "313797" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "instance": "10.142.0.107:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "instance": "10.142.0.77:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "687474125824" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "1749.1" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 ERROR [http_cmd]: ERROR 03:08:03 ERROR --- status code --- 03:08:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:08:03 ERROR --- http response --- 03:08:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:08:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "0.023" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "1748.6333333333332" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "ds-idrepo-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "ds-cts-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "28.53333333333333" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "ds-idrepo-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "ds-cts-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "0.0005749916666666667" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "instance": "10.142.0.107:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "instance": "10.142.0.77:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "0.18286666666665544" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG "data": { 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "ds-idrepo-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:08:03 DEBUG 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG "pod": "ds-cts-1" 03:08:03 DEBUG { 03:08:03 DEBUG }, 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "value": [ 03:08:03 DEBUG "data": { 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "0" 03:08:03 DEBUG "result": [ 03:08:03 DEBUG ] 03:08:03 DEBUG { 03:08:03 DEBUG } 03:08:03 DEBUG "metric": { 03:08:03 DEBUG ] 03:08:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:08:03 DEBUG } 03:08:03 DEBUG }, 03:08:03 DEBUG } 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "idm-b85f9d978-dz57p" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "9805.273985412381" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "ds-idrepo-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "ds-idrepo-0" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "0" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "instance": "10.142.0.107:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "instance": "10.142.0.77:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "51290338.7" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "idm-b85f9d978-dz57p" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "612.8296240882738" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "instance": "10.142.0.107:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "instance": "10.142.0.77:9100" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "51286985.233333334" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974083" 03:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974083" 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "ds-idrepo-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "ds-cts-1" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "0" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG } 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "1748.1" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:03 INFO [http_cmd]: http status code OK 03:08:03 DEBUG --- status code --- 03:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:03 DEBUG --- http response --- 03:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:03 DEBUG ----- output ----- 03:08:03 DEBUG { 03:08:03 DEBUG "status": "success", 03:08:03 DEBUG "data": { 03:08:03 DEBUG "resultType": "vector", 03:08:03 DEBUG "result": [ 03:08:03 DEBUG { 03:08:03 DEBUG "metric": { 03:08:03 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG [...] 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG 03:08:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:08:03 DEBUG }, 03:08:03 DEBUG "value": [ 03:08:03 DEBUG 1712974083, 03:08:03 DEBUG "5.301741221033189" 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG ] 03:08:03 DEBUG } 03:08:03 DEBUG } 03:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974084" 03:08:04 INFO [http_cmd]: http status code OK 03:08:04 DEBUG --- status code --- 03:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:04 DEBUG --- http response --- 03:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:04 DEBUG ----- output ----- 03:08:04 DEBUG { 03:08:04 DEBUG "status": "success", 03:08:04 DEBUG "data": { 03:08:04 DEBUG "resultType": "vector", 03:08:04 DEBUG "result": [ 03:08:04 DEBUG { 03:08:04 DEBUG "metric": { 03:08:04 DEBUG "pod": "ds-idrepo-1" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG [...] 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG "pod": "ds-idrepo-0" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 1712974084, 03:08:04 DEBUG "0" 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG } 03:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974084" 03:08:04 INFO [http_cmd]: http status code OK 03:08:04 DEBUG --- status code --- 03:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:04 DEBUG --- http response --- 03:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:08:04 DEBUG ----- output ----- 03:08:04 DEBUG { 03:08:04 DEBUG "status": "success", 03:08:04 DEBUG "data": { 03:08:04 DEBUG "resultType": "vector", 03:08:04 DEBUG "result": [ 03:08:04 DEBUG { 03:08:04 DEBUG "metric": { 03:08:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG [...] 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 1712974084, 03:08:04 DEBUG "0" 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG } 03:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974084" 03:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974084" 03:08:04 INFO [http_cmd]: http status code OK 03:08:04 DEBUG --- status code --- 03:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:04 DEBUG --- http response --- 03:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:08:04 DEBUG ----- output ----- 03:08:04 DEBUG { 03:08:04 DEBUG "status": "success", 03:08:04 DEBUG "data": { 03:08:04 DEBUG "resultType": "vector", 03:08:04 DEBUG "result": [ 03:08:04 DEBUG { 03:08:04 DEBUG "metric": { 03:08:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG [...] 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 1712974084, 03:08:04 DEBUG "0.0028663923695525285" 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG } 03:08:04 INFO [http_cmd]: http status code OK 03:08:04 DEBUG --- status code --- 03:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:04 DEBUG --- http response --- 03:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:04 DEBUG ----- output ----- 03:08:04 DEBUG { 03:08:04 DEBUG "status": "success", 03:08:04 DEBUG "data": { 03:08:04 DEBUG "resultType": "vector", 03:08:04 DEBUG "result": [ 03:08:04 DEBUG { 03:08:04 DEBUG "metric": { 03:08:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG [...] 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 1712974084, 03:08:04 DEBUG "6236860416" 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG } 03:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974084" 03:08:04 INFO [http_cmd]: http status code OK 03:08:04 DEBUG --- status code --- 03:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:04 DEBUG --- http response --- 03:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:08:04 DEBUG ----- output ----- 03:08:04 DEBUG { 03:08:04 DEBUG "status": "success", 03:08:04 DEBUG "data": { 03:08:04 DEBUG "resultType": "vector", 03:08:04 DEBUG "result": [ 03:08:04 DEBUG { 03:08:04 DEBUG "metric": { 03:08:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG [...] 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 1712974084, 03:08:04 DEBUG "0" 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG } 03:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974084" 03:08:04 INFO [http_cmd]: http status code OK 03:08:04 DEBUG --- status code --- 03:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:08:04 DEBUG --- http response --- 03:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:08:04 DEBUG ----- output ----- 03:08:04 DEBUG { 03:08:04 DEBUG "status": "success", 03:08:04 DEBUG "data": { 03:08:04 DEBUG "resultType": "vector", 03:08:04 DEBUG "result": [ 03:08:04 DEBUG { 03:08:04 DEBUG "metric": { 03:08:04 DEBUG "instance": "10.142.0.107:9100" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG [...] 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG 03:08:04 DEBUG "instance": "10.142.0.77:9100" 03:08:04 DEBUG }, 03:08:04 DEBUG "value": [ 03:08:04 DEBUG 1712974084, 03:08:04 DEBUG "0.06033333333333364" 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG ] 03:08:04 DEBUG } 03:08:04 DEBUG } 03:08:17 INFO 03:08:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:08:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:08:17 INFO [loop_until]: OK (rc = 0) 03:08:17 DEBUG --- stdout --- 03:08:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11839m 8581Mi am-6d96779b5d-qbg7l 9861m 8483Mi am-6d96779b5d-zn69c 11585m 8482Mi ds-cts-0 10595m 13518Mi ds-cts-1 11058m 13512Mi ds-cts-2 10792m 13519Mi ds-idrepo-0 7931m 5594Mi ds-idrepo-1 4263m 5523Mi ds-idrepo-2 4271m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1120Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5158m 4193Mi 03:08:17 DEBUG --- stderr --- 03:08:17 DEBUG 03:08:24 INFO 03:08:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:08:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:08:24 INFO [loop_until]: OK (rc = 0) 03:08:24 DEBUG --- stdout --- 03:08:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4739Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1873Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10256m 34% 10364Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14073m 47% 3102Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12164m 40% 10287Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 128m 0% 4317Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12790m 42% 10427Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4425m 27% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10861m 68% 14428Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11106m 69% 14396Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4242m 26% 6698Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10996m 69% 14433Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8283m 52% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5209m 32% 7934Mi 13% 03:08:24 DEBUG --- stderr --- 03:08:24 DEBUG 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "ds-idrepo-1" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "ds-idrepo-0" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "0" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "1751.0666666666666" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "314172" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974143" 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "instance": "10.142.0.107:9100" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "instance": "10.142.0.77:9100" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "690265334784" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 ERROR [http_cmd]: ERROR 03:09:03 ERROR --- status code --- 03:09:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:09:03 ERROR --- http response --- 03:09:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:09:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "ds-idrepo-1" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "ds-cts-1" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "27.866666666666667" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "0.023" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "1748.0666666666666" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "1748" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974143" 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "pod": "ds-idrepo-1" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "pod": "ds-cts-1" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "0.0001" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: http status code OK 03:09:03 DEBUG --- status code --- 03:09:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:03 DEBUG --- http response --- 03:09:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:03 DEBUG ----- output ----- 03:09:03 DEBUG { 03:09:03 DEBUG "status": "success", 03:09:03 DEBUG "data": { 03:09:03 DEBUG "resultType": "vector", 03:09:03 DEBUG "result": [ 03:09:03 DEBUG { 03:09:03 DEBUG "metric": { 03:09:03 DEBUG "instance": "10.142.0.107:9100" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG [...] 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG 03:09:03 DEBUG "instance": "10.142.0.77:9100" 03:09:03 DEBUG }, 03:09:03 DEBUG "value": [ 03:09:03 DEBUG 1712974143, 03:09:03 DEBUG "0.17983333333332893" 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG ] 03:09:03 DEBUG } 03:09:03 DEBUG } 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974143" 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974143" 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974144" 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG { 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "data": { 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG "result": [ 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG { 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "ds-idrepo-1" 03:09:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:09:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:09:04 DEBUG }, 03:09:04 DEBUG }, 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG "value": [ 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG [...] 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-cts-1" 03:09:04 DEBUG "pod": "ds-cts-1" 03:09:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:09:04 DEBUG }, 03:09:04 DEBUG }, 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG "value": [ 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974143, 03:09:04 DEBUG 1712974143, 03:09:04 DEBUG 1712974143, 03:09:04 DEBUG "0" 03:09:04 DEBUG "1209023.6966824646" 03:09:04 DEBUG "5.267887526711307" 03:09:04 DEBUG ] 03:09:04 DEBUG ] 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG ] 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "instance": "10.142.0.107:9100" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "instance": "10.142.0.77:9100" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "51186857.166666664" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974144" 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "ds-idrepo-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-idrepo-0" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "0" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "instance": "10.142.0.107:9100" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "instance": "10.142.0.77:9100" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "51190235.43333334" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-cts-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "44848166.82464455" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "1748.5666666666666" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "ds-idrepo-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-cts-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "0" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "ds-idrepo-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-idrepo-0" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "0" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974144" 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974144" 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-cts-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "29630483.412322275" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "0" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "pod": "ds-cts-1" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "10.997560756101947" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "6236659712" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974144" 03:09:04 INFO [http_cmd]: http status code OK 03:09:04 DEBUG --- status code --- 03:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:09:04 DEBUG --- http response --- 03:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:09:04 DEBUG ----- output ----- 03:09:04 DEBUG { 03:09:04 DEBUG "status": "success", 03:09:04 DEBUG "data": { 03:09:04 DEBUG "resultType": "vector", 03:09:04 DEBUG "result": [ 03:09:04 DEBUG { 03:09:04 DEBUG "metric": { 03:09:04 DEBUG "instance": "10.142.0.107:9100" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG [...] 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG 03:09:04 DEBUG "instance": "10.142.0.77:9100" 03:09:04 DEBUG }, 03:09:04 DEBUG "value": [ 03:09:04 DEBUG 1712974144, 03:09:04 DEBUG "0.05433333333333483" 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG ] 03:09:04 DEBUG } 03:09:04 DEBUG } 03:09:17 INFO 03:09:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:09:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:09:17 INFO [loop_until]: OK (rc = 0) 03:09:17 DEBUG --- stdout --- 03:09:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12271m 8574Mi am-6d96779b5d-qbg7l 9912m 8484Mi am-6d96779b5d-zn69c 11876m 8483Mi ds-cts-0 10345m 13518Mi ds-cts-1 10807m 12747Mi ds-cts-2 10592m 13585Mi ds-idrepo-0 7857m 5594Mi ds-idrepo-1 4322m 5523Mi ds-idrepo-2 4192m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 3m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5102m 4196Mi 03:09:17 DEBUG --- stderr --- 03:09:17 DEBUG 03:09:24 INFO 03:09:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:09:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:09:24 INFO [loop_until]: OK (rc = 0) 03:09:24 DEBUG --- stdout --- 03:09:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 4737Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1856Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10368m 34% 10357Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14114m 47% 3111Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12082m 40% 10285Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4336Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12882m 43% 10432Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4353m 27% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10733m 67% 14483Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11064m 69% 14411Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4243m 26% 6701Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10895m 68% 14447Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8252m 51% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5374m 33% 7933Mi 13% 03:09:24 DEBUG --- stderr --- 03:09:24 DEBUG 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "pod": "ds-idrepo-1" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "pod": "ds-idrepo-0" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "0" 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "1751.3999999999999" 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "314244" 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974203" 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974203" 03:10:03 ERROR [http_cmd]: ERROR 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 ERROR --- status code --- 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 ERROR --- http response --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "instance": "10.142.0.107:9100" 03:10:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:03 DEBUG }, 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "instance": "10.142.0.77:9100" 03:10:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:03 DEBUG }, 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "692554132480" 03:10:03 DEBUG "0.023" 03:10:03 DEBUG ] 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "pod": "ds-idrepo-1" 03:10:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:03 DEBUG }, 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "pod": "ds-cts-1" 03:10:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:03 DEBUG }, 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "28.9" 03:10:03 DEBUG "1746.8" 03:10:03 DEBUG ] 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "1747" 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "pod": "ds-idrepo-1" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "pod": "ds-cts-1" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "0.00009999999999999998" 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974203" 03:10:03 INFO [http_cmd]: http status code OK 03:10:03 DEBUG --- status code --- 03:10:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:03 DEBUG --- http response --- 03:10:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:03 DEBUG ----- output ----- 03:10:03 DEBUG { 03:10:03 DEBUG "status": "success", 03:10:03 DEBUG "data": { 03:10:03 DEBUG "resultType": "vector", 03:10:03 DEBUG "result": [ 03:10:03 DEBUG { 03:10:03 DEBUG "metric": { 03:10:03 DEBUG "instance": "10.142.0.107:9100" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG [...] 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG 03:10:03 DEBUG "instance": "10.142.0.77:9100" 03:10:03 DEBUG }, 03:10:03 DEBUG "value": [ 03:10:03 DEBUG 1712974203, 03:10:03 DEBUG "0.1621333333333406" 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG ] 03:10:03 DEBUG } 03:10:03 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "ds-idrepo-1" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "ds-cts-1" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974204" 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "instance": "10.142.0.107:9100" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "instance": "10.142.0.77:9100" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "51464710.03333333" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "instance": "10.142.0.107:9100" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "instance": "10.142.0.77:9100" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "51467114.86666667" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "5.2468550526668425" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "ds-idrepo-1" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "ds-idrepo-0" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "ds-idrepo-1" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "ds-cts-1" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "1746.6333333333332" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "ds-idrepo-1" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "ds-idrepo-0" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:10:04 DEBUG }, 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG { 03:10:04 DEBUG [...] 03:10:04 DEBUG [...] 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "data": { 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "result": [ 03:10:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:10:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:10:04 DEBUG { 03:10:04 DEBUG }, 03:10:04 DEBUG }, 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "value": [ 03:10:04 DEBUG "value": [ 03:10:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG }, 03:10:04 DEBUG "0" 03:10:04 DEBUG "0.0018126790532914947" 03:10:04 DEBUG "value": [ 03:10:04 DEBUG ] 03:10:04 DEBUG ] 03:10:04 DEBUG 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 DEBUG 03:10:04 DEBUG ] 03:10:04 DEBUG ] 03:10:04 DEBUG 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 DEBUG [...] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "6237126656" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974204" 03:10:04 INFO [http_cmd]: http status code OK 03:10:04 DEBUG --- status code --- 03:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:10:04 DEBUG --- http response --- 03:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:10:04 DEBUG ----- output ----- 03:10:04 DEBUG { 03:10:04 DEBUG "status": "success", 03:10:04 DEBUG "data": { 03:10:04 DEBUG "resultType": "vector", 03:10:04 DEBUG "result": [ 03:10:04 DEBUG { 03:10:04 DEBUG "metric": { 03:10:04 DEBUG "instance": "10.142.0.107:9100" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG [...] 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG 03:10:04 DEBUG "instance": "10.142.0.77:9100" 03:10:04 DEBUG }, 03:10:04 DEBUG "value": [ 03:10:04 DEBUG 1712974204, 03:10:04 DEBUG "0.04600000000000221" 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG ] 03:10:04 DEBUG } 03:10:04 DEBUG } 03:10:17 INFO 03:10:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:10:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:10:17 INFO [loop_until]: OK (rc = 0) 03:10:17 DEBUG --- stdout --- 03:10:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11928m 8571Mi am-6d96779b5d-qbg7l 9677m 8485Mi am-6d96779b5d-zn69c 11752m 8480Mi ds-cts-0 11026m 13517Mi ds-cts-1 10836m 13216Mi ds-cts-2 10411m 13520Mi ds-idrepo-0 8022m 5594Mi ds-idrepo-1 4354m 5523Mi ds-idrepo-2 4137m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1121Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5144m 4193Mi 03:10:17 DEBUG --- stderr --- 03:10:17 DEBUG 03:10:24 INFO 03:10:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:10:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:10:24 INFO [loop_until]: OK (rc = 0) 03:10:24 DEBUG --- stdout --- 03:10:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4743Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1863Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9911m 33% 10368Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14006m 46% 3096Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12151m 40% 10291Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 4339Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12629m 42% 10427Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4361m 27% 6304Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10610m 66% 14419Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11056m 69% 14154Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4190m 26% 6701Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10997m 69% 14521Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8243m 51% 6388Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5346m 33% 7931Mi 13% 03:10:24 DEBUG --- stderr --- 03:10:24 DEBUG 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "ds-idrepo-1" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "ds-idrepo-0" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "0" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "1724.9333333333334" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "314189" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "0.023" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974263" 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "instance": "10.142.0.107:9100" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "instance": "10.142.0.77:9100" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "694759742464" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 ERROR [http_cmd]: ERROR 03:11:03 ERROR --- status code --- 03:11:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:11:03 ERROR --- http response --- 03:11:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:11:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "1750.6" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "ds-idrepo-1" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "ds-cts-1" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "29.566666666666666" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974263" 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "ds-idrepo-1" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "ds-cts-1" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "0.0021362" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:03 INFO [http_cmd]: http status code OK 03:11:03 DEBUG --- status code --- 03:11:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:03 DEBUG --- http response --- 03:11:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:03 DEBUG ----- output ----- 03:11:03 DEBUG { 03:11:03 DEBUG "status": "success", 03:11:03 DEBUG "data": { 03:11:03 DEBUG "resultType": "vector", 03:11:03 DEBUG "result": [ 03:11:03 DEBUG { 03:11:03 DEBUG "metric": { 03:11:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG [...] 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG 03:11:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:03 DEBUG }, 03:11:03 DEBUG "value": [ 03:11:03 DEBUG 1712974263, 03:11:03 DEBUG "1747.4666666666667" 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG ] 03:11:03 DEBUG } 03:11:03 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "instance": "10.142.0.107:9100" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "instance": "10.142.0.77:9100" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0.18003333333333746" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974264" 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG { 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "ds-idrepo-1" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "ds-cts-1" 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:04 DEBUG }, 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG "value": [ 03:11:04 DEBUG { 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "0" 03:11:04 DEBUG "data": { 03:11:04 DEBUG ] 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG } 03:11:04 DEBUG "result": [ 03:11:04 DEBUG ] 03:11:04 DEBUG { 03:11:04 DEBUG } 03:11:04 DEBUG "metric": { 03:11:04 DEBUG } 03:11:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "5.223013355884218" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "instance": "10.142.0.107:9100" 03:11:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:11:04 DEBUG }, 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "instance": "10.142.0.77:9100" 03:11:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:11:04 DEBUG }, 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "50831953.93333334" 03:11:04 DEBUG "0" 03:11:04 DEBUG ] 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "ds-idrepo-1" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "ds-idrepo-0" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "instance": "10.142.0.107:9100" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "instance": "10.142.0.77:9100" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "50835474.766666666" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG { 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "ds-idrepo-1" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "ds-cts-1" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "1748.2666666666667" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "ds-idrepo-1" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "ds-idrepo-0" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974264" 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG { 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:11:04 DEBUG "data": { 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG }, 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "value": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:11:04 DEBUG 03:11:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:11:04 DEBUG }, 03:11:04 DEBUG 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG [...] 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG }, 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG 03:11:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:11:04 DEBUG "0" 03:11:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:11:04 DEBUG }, 03:11:04 DEBUG ] 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG } 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG ] 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0" 03:11:04 DEBUG } 03:11:04 DEBUG "6237380608" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974264" 03:11:04 INFO [http_cmd]: http status code OK 03:11:04 DEBUG --- status code --- 03:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:11:04 DEBUG --- http response --- 03:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:11:04 DEBUG ----- output ----- 03:11:04 DEBUG { 03:11:04 DEBUG "status": "success", 03:11:04 DEBUG "data": { 03:11:04 DEBUG "resultType": "vector", 03:11:04 DEBUG "result": [ 03:11:04 DEBUG { 03:11:04 DEBUG "metric": { 03:11:04 DEBUG "instance": "10.142.0.107:9100" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG [...] 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG 03:11:04 DEBUG "instance": "10.142.0.77:9100" 03:11:04 DEBUG }, 03:11:04 DEBUG "value": [ 03:11:04 DEBUG 1712974264, 03:11:04 DEBUG "0.0610000000000016" 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG ] 03:11:04 DEBUG } 03:11:04 DEBUG } 03:11:17 INFO 03:11:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:11:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:11:17 INFO [loop_until]: OK (rc = 0) 03:11:17 DEBUG --- stdout --- 03:11:17 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12341m 8572Mi am-6d96779b5d-qbg7l 10068m 8483Mi am-6d96779b5d-zn69c 11752m 8486Mi ds-cts-0 10540m 13518Mi ds-cts-1 10516m 13514Mi ds-cts-2 10375m 13518Mi ds-idrepo-0 8216m 5594Mi ds-idrepo-1 4217m 5523Mi ds-idrepo-2 4224m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5246m 4195Mi 03:11:17 DEBUG --- stderr --- 03:11:17 DEBUG 03:11:24 INFO 03:11:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:11:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:11:24 INFO [loop_until]: OK (rc = 0) 03:11:24 DEBUG --- stdout --- 03:11:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4760Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 1861Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10502m 35% 10361Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14096m 47% 3113Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12116m 40% 10301Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 129m 0% 4366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12786m 42% 10422Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4211m 26% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10736m 67% 14441Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10640m 66% 14391Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4222m 26% 6702Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10532m 66% 14452Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8259m 51% 6390Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5379m 33% 7937Mi 13% 03:11:24 DEBUG --- stderr --- 03:11:24 DEBUG 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "ds-idrepo-1" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "ds-idrepo-0" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "0" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "1748.1333333333332" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974323" 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "314082.5" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "0.023" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "instance": "10.142.0.107:9100" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "instance": "10.142.0.77:9100" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "697471001600" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974323" 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "1743.7666666666667" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 ERROR [http_cmd]: ERROR 03:12:03 ERROR --- status code --- 03:12:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:12:03 ERROR --- http response --- 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:12:03 DEBUG --- status code --- 03:12:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "ds-idrepo-1" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "ds-cts-1" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "29.9" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974323" 03:12:03 INFO [http_cmd]: http status code OK 03:12:03 DEBUG --- status code --- 03:12:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:03 DEBUG --- http response --- 03:12:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:12:03 DEBUG ----- output ----- 03:12:03 DEBUG { 03:12:03 DEBUG "status": "success", 03:12:03 DEBUG "data": { 03:12:03 DEBUG "resultType": "vector", 03:12:03 DEBUG "result": [ 03:12:03 DEBUG { 03:12:03 DEBUG "metric": { 03:12:03 DEBUG "pod": "ds-idrepo-1" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG [...] 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG 03:12:03 DEBUG "pod": "ds-cts-1" 03:12:03 DEBUG }, 03:12:03 DEBUG "value": [ 03:12:03 DEBUG 1712974323, 03:12:03 DEBUG "0.0010800583333333334" 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG ] 03:12:03 DEBUG } 03:12:03 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "1743.7" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "instance": "10.142.0.107:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "instance": "10.142.0.77:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0.20023333333333113" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "5.289022594925458" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-cts-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:12:04 DEBUG "instance": "10.142.0.107:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG "instance": "10.142.0.77:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "4733.6759830905385" 03:12:04 DEBUG "51398638.33333333" 03:12:04 DEBUG ] 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-idrepo-0" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "1745.3666666666666" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "instance": "10.142.0.107:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "instance": "10.142.0.77:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "51404013.03333333" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "3114.2605151911434" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-cts-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-idrepo-0" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "4.305200688604505" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "ds-idrepo-1" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "instance": "10.142.0.107:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "instance": "10.142.0.77:9100" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0.07766666666666683" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974324" 03:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974324" 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "0" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:04 INFO [http_cmd]: http status code OK 03:12:04 DEBUG --- status code --- 03:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:12:04 DEBUG --- http response --- 03:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:12:04 DEBUG ----- output ----- 03:12:04 DEBUG { 03:12:04 DEBUG "status": "success", 03:12:04 DEBUG "data": { 03:12:04 DEBUG "resultType": "vector", 03:12:04 DEBUG "result": [ 03:12:04 DEBUG { 03:12:04 DEBUG "metric": { 03:12:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG [...] 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG 03:12:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:12:04 DEBUG }, 03:12:04 DEBUG "value": [ 03:12:04 DEBUG 1712974324, 03:12:04 DEBUG "6237974528" 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG ] 03:12:04 DEBUG } 03:12:04 DEBUG } 03:12:17 INFO 03:12:17 INFO [loop_until]: kubectl --namespace=xlou top pods 03:12:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:12:18 INFO [loop_until]: OK (rc = 0) 03:12:18 DEBUG --- stdout --- 03:12:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11782m 8572Mi am-6d96779b5d-qbg7l 9817m 8483Mi am-6d96779b5d-zn69c 11541m 8491Mi ds-cts-0 10517m 13440Mi ds-cts-1 11359m 13636Mi ds-cts-2 10572m 13485Mi ds-idrepo-0 8085m 5594Mi ds-idrepo-1 4331m 5523Mi ds-idrepo-2 4121m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5276m 4195Mi 03:12:18 DEBUG --- stderr --- 03:12:18 DEBUG 03:12:24 INFO 03:12:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:12:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:12:24 INFO [loop_until]: OK (rc = 0) 03:12:24 DEBUG --- stdout --- 03:12:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4753Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10205m 34% 10359Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13956m 46% 3103Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11953m 40% 10295Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12735m 42% 10420Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4399m 27% 6308Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10638m 66% 14448Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11204m 70% 14466Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4224m 26% 6702Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10491m 66% 14436Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8255m 51% 6390Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5410m 34% 7936Mi 13% 03:12:24 DEBUG --- stderr --- 03:12:24 DEBUG 03:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974383" 03:13:03 INFO [http_cmd]: http status code OK 03:13:03 DEBUG --- status code --- 03:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:03 DEBUG --- http response --- 03:13:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974383" 03:13:03 INFO [http_cmd]: http status code OK 03:13:03 DEBUG --- status code --- 03:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:03 DEBUG --- http response --- 03:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:03 DEBUG ----- output ----- 03:13:03 DEBUG { 03:13:03 DEBUG "status": "success", 03:13:03 DEBUG "data": { 03:13:03 DEBUG "resultType": "vector", 03:13:03 DEBUG "result": [ 03:13:03 DEBUG { 03:13:03 DEBUG "metric": { 03:13:03 DEBUG "pod": "ds-idrepo-1" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG [...] 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG "pod": "ds-idrepo-0" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 1712974383, 03:13:03 DEBUG "0" 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG } 03:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974383" 03:13:03 INFO [http_cmd]: http status code OK 03:13:03 DEBUG --- status code --- 03:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:03 DEBUG --- http response --- 03:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:03 DEBUG ----- output ----- 03:13:03 DEBUG { 03:13:03 DEBUG "status": "success", 03:13:03 DEBUG "data": { 03:13:03 DEBUG "resultType": "vector", 03:13:03 DEBUG "result": [ 03:13:03 DEBUG { 03:13:03 DEBUG "metric": { 03:13:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG [...] 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 1712974383, 03:13:03 DEBUG "1745.8" 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG } 03:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974383" 03:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974383" 03:13:03 INFO [http_cmd]: http status code OK 03:13:03 INFO [http_cmd]: http status code OK 03:13:03 DEBUG --- status code --- 03:13:03 DEBUG --- status code --- 03:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:03 DEBUG --- http response --- 03:13:03 DEBUG --- http response --- 03:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:03 DEBUG ----- output ----- 03:13:03 DEBUG { 03:13:03 DEBUG "status": "success", 03:13:03 DEBUG "data": { 03:13:03 DEBUG "resultType": "vector", 03:13:03 DEBUG "result": [ 03:13:03 DEBUG { 03:13:03 DEBUG "metric": { 03:13:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG [...] 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 1712974383, 03:13:03 DEBUG "0.023" 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG } 03:13:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:03 DEBUG ----- output ----- 03:13:03 DEBUG { 03:13:03 DEBUG "status": "success", 03:13:03 DEBUG "data": { 03:13:03 DEBUG "resultType": "vector", 03:13:03 DEBUG "result": [ 03:13:03 DEBUG { 03:13:03 DEBUG "metric": { 03:13:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG [...] 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG 03:13:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:03 DEBUG }, 03:13:03 DEBUG "value": [ 03:13:03 DEBUG 1712974383, 03:13:03 DEBUG "313529.5" 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG ] 03:13:03 DEBUG } 03:13:03 DEBUG } 03:13:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974383" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "instance": "10.142.0.107:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "instance": "10.142.0.77:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974383, 03:13:04 DEBUG "699765453824" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "1726.3333333333333" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 ERROR [http_cmd]: ERROR 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 ERROR --- status code --- 03:13:04 DEBUG --- status code --- 03:13:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 ERROR --- http response --- 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:13:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "ds-idrepo-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-cts-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "1727.1333333333332" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "ds-idrepo-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-cts-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "29.099999999999998" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "instance": "10.142.0.107:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "instance": "10.142.0.77:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0.06120000000000611" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "5.309337464934" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "instance": "10.142.0.107:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "instance": "10.142.0.77:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "51124101.166666664" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "ds-idrepo-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-cts-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-cts-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "ds-idrepo-1" 03:13:04 DEBUG "instance": "10.142.0.107:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-idrepo-0" 03:13:04 DEBUG "instance": "10.142.0.77:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG "51120651.5" 03:13:04 DEBUG ] 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "1727.1333333333332" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG { 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "ds-idrepo-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:13:04 DEBUG 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG 03:13:04 DEBUG { 03:13:04 DEBUG 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG [...] 03:13:04 DEBUG "data": { 03:13:04 DEBUG 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG 03:13:04 DEBUG "result": [ 03:13:04 DEBUG 03:13:04 DEBUG { 03:13:04 DEBUG "pod": "ds-cts-1" 03:13:04 DEBUG "metric": { 03:13:04 DEBUG }, 03:13:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:13:04 DEBUG "value": [ 03:13:04 DEBUG }, 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG "0" 03:13:04 DEBUG 03:13:04 DEBUG ] 03:13:04 DEBUG 03:13:04 DEBUG } 03:13:04 DEBUG 03:13:04 DEBUG ] 03:13:04 DEBUG [...] 03:13:04 DEBUG } 03:13:04 DEBUG 03:13:04 DEBUG } 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "ds-idrepo-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-idrepo-0" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "6238715904" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "ds-cts-1" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG { 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "instance": "10.142.0.107:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "instance": "10.142.0.77:9100" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0.028333333333333852" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974384" 03:13:04 INFO [http_cmd]: http status code OK 03:13:04 DEBUG --- status code --- 03:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:13:04 DEBUG --- http response --- 03:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:13:04 DEBUG ----- output ----- 03:13:04 DEBUG { 03:13:04 DEBUG "status": "success", 03:13:04 DEBUG "data": { 03:13:04 DEBUG "resultType": "vector", 03:13:04 DEBUG "result": [ 03:13:04 DEBUG { 03:13:04 DEBUG "metric": { 03:13:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG [...] 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG 03:13:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:13:04 DEBUG }, 03:13:04 DEBUG "value": [ 03:13:04 DEBUG 1712974384, 03:13:04 DEBUG "0" 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG ] 03:13:04 DEBUG } 03:13:04 DEBUG } 03:13:18 INFO 03:13:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:13:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:13:18 INFO [loop_until]: OK (rc = 0) 03:13:18 DEBUG --- stdout --- 03:13:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12078m 8574Mi am-6d96779b5d-qbg7l 9739m 8483Mi am-6d96779b5d-zn69c 11684m 8484Mi ds-cts-0 10643m 13518Mi ds-cts-1 11251m 13233Mi ds-cts-2 10397m 13594Mi ds-idrepo-0 8040m 5593Mi ds-idrepo-1 4337m 5523Mi ds-idrepo-2 4287m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5175m 4195Mi 03:13:18 DEBUG --- stderr --- 03:13:18 DEBUG 03:13:24 INFO 03:13:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:13:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:13:24 INFO [loop_until]: OK (rc = 0) 03:13:24 DEBUG --- stdout --- 03:13:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4746Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1853Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10328m 34% 10360Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14156m 47% 3103Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12469m 41% 10296Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12871m 43% 10433Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4415m 27% 6308Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10833m 68% 14522Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11259m 70% 14124Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4303m 27% 6707Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 11033m 69% 14447Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8331m 52% 6390Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5389m 33% 7936Mi 13% 03:13:24 DEBUG --- stderr --- 03:13:24 DEBUG 03:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974443" 03:14:03 INFO [http_cmd]: http status code OK 03:14:03 DEBUG --- status code --- 03:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:03 DEBUG --- http response --- 03:14:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974443" 03:14:03 INFO [http_cmd]: http status code OK 03:14:03 DEBUG --- status code --- 03:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:03 DEBUG --- http response --- 03:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:03 DEBUG ----- output ----- 03:14:03 DEBUG { 03:14:03 DEBUG "status": "success", 03:14:03 DEBUG "data": { 03:14:03 DEBUG "resultType": "vector", 03:14:03 DEBUG "result": [ 03:14:03 DEBUG { 03:14:03 DEBUG "metric": { 03:14:03 DEBUG "pod": "ds-idrepo-1" 03:14:03 DEBUG }, 03:14:03 DEBUG "value": [ 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG [...] 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG "pod": "ds-idrepo-0" 03:14:03 DEBUG }, 03:14:03 DEBUG "value": [ 03:14:03 DEBUG 1712974443, 03:14:03 DEBUG "0" 03:14:03 DEBUG ] 03:14:03 DEBUG } 03:14:03 DEBUG ] 03:14:03 DEBUG } 03:14:03 DEBUG } 03:14:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974443" 03:14:03 INFO [http_cmd]: http status code OK 03:14:03 DEBUG --- status code --- 03:14:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:03 DEBUG --- http response --- 03:14:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:03 DEBUG ----- output ----- 03:14:03 DEBUG { 03:14:03 DEBUG "status": "success", 03:14:03 DEBUG "data": { 03:14:03 DEBUG "resultType": "vector", 03:14:03 DEBUG "result": [ 03:14:03 DEBUG { 03:14:03 DEBUG "metric": { 03:14:03 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:03 DEBUG }, 03:14:03 DEBUG "value": [ 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG [...] 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG 03:14:03 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:03 DEBUG }, 03:14:03 DEBUG "value": [ 03:14:03 DEBUG 1712974443, 03:14:03 DEBUG "1751.6333333333332" 03:14:03 DEBUG ] 03:14:03 DEBUG } 03:14:03 DEBUG ] 03:14:03 DEBUG } 03:14:03 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "313337" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0.023" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "ds-idrepo-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "ds-cts-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "27.766666666666666" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "instance": "10.142.0.107:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "instance": "10.142.0.77:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "702120496128" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "1745.9333333333334" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 ERROR [http_cmd]: ERROR 03:14:04 ERROR --- status code --- 03:14:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:14:04 ERROR --- http response --- 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG ----- output ----- 03:14:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG { 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "1744.2333333333333" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "ds-idrepo-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "ds-cts-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0.0006707638888888888" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "5.263973190788568" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "instance": "10.142.0.107:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "instance": "10.142.0.77:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0.1832000000000112" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "instance": "10.142.0.107:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "instance": "10.142.0.77:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "51383072.83333333" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG "result": [ 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG { 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG "metric": { 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG "pod": "ds-idrepo-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "ds-idrepo-0" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "ds-idrepo-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "ds-cts-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "instance": "10.142.0.107:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "instance": "10.142.0.77:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "51377550.46666667" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "1744.7" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "ds-idrepo-1" 03:14:04 DEBUG }, 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:14:04 DEBUG "value": [ 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG 03:14:04 DEBUG { 03:14:04 DEBUG 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG 03:14:04 DEBUG "data": { 03:14:04 DEBUG [...] 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG 03:14:04 DEBUG "result": [ 03:14:04 DEBUG 03:14:04 DEBUG { 03:14:04 DEBUG 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "ds-cts-1" 03:14:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:14:04 DEBUG }, 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG 03:14:04 DEBUG "0" 03:14:04 DEBUG 03:14:04 DEBUG ] 03:14:04 DEBUG 03:14:04 DEBUG } 03:14:04 DEBUG [...] 03:14:04 DEBUG ] 03:14:04 DEBUG 03:14:04 DEBUG } 03:14:04 DEBUG 03:14:04 DEBUG } 03:14:04 DEBUG 03:14:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "ds-idrepo-1" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "ds-idrepo-0" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "6240358400" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "instance": "10.142.0.107:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "instance": "10.142.0.77:9100" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0.06400000000000124" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974444" 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:04 INFO [http_cmd]: http status code OK 03:14:04 DEBUG --- status code --- 03:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:14:04 DEBUG --- http response --- 03:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:14:04 DEBUG ----- output ----- 03:14:04 DEBUG { 03:14:04 DEBUG "status": "success", 03:14:04 DEBUG "data": { 03:14:04 DEBUG "resultType": "vector", 03:14:04 DEBUG "result": [ 03:14:04 DEBUG { 03:14:04 DEBUG "metric": { 03:14:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG [...] 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG 03:14:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:14:04 DEBUG }, 03:14:04 DEBUG "value": [ 03:14:04 DEBUG 1712974444, 03:14:04 DEBUG "0" 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG ] 03:14:04 DEBUG } 03:14:04 DEBUG } 03:14:18 INFO 03:14:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:14:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:14:18 INFO [loop_until]: OK (rc = 0) 03:14:18 DEBUG --- stdout --- 03:14:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12317m 8574Mi am-6d96779b5d-qbg7l 9708m 8484Mi am-6d96779b5d-zn69c 11582m 8487Mi ds-cts-0 10768m 13519Mi ds-cts-1 10713m 13516Mi ds-cts-2 10780m 13518Mi ds-idrepo-0 8165m 5594Mi ds-idrepo-1 4335m 5523Mi ds-idrepo-2 4269m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5124m 4194Mi 03:14:18 DEBUG --- stderr --- 03:14:18 DEBUG 03:14:24 INFO 03:14:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:14:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:14:24 INFO [loop_until]: OK (rc = 0) 03:14:24 DEBUG --- stdout --- 03:14:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 4741Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1857Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10201m 34% 10359Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14003m 46% 3093Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12096m 40% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4374Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12752m 42% 10424Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4423m 27% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10710m 67% 14442Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11100m 69% 14406Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4131m 25% 6705Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10683m 67% 14436Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8243m 51% 6388Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5367m 33% 7937Mi 13% 03:14:24 DEBUG --- stderr --- 03:14:24 DEBUG 03:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974503" 03:15:03 INFO [http_cmd]: http status code OK 03:15:03 DEBUG --- status code --- 03:15:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:03 DEBUG --- http response --- 03:15:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:15:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974503" 03:15:03 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-idrepo-0" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974503, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "1760.5333333333333" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "312972" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0.023" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-cts-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0.0006377694444444444" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:15:04 DEBUG "result": [ 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG "data": { 03:15:04 DEBUG }, 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "value": [ 03:15:04 DEBUG "result": [ 03:15:04 DEBUG 03:15:04 DEBUG { 03:15:04 DEBUG 03:15:04 DEBUG "metric": { 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG [...] 03:15:04 DEBUG }, 03:15:04 DEBUG 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG 03:15:04 DEBUG }, 03:15:04 DEBUG [...] 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG 03:15:04 DEBUG "1749.9666666666667" 03:15:04 DEBUG 03:15:04 DEBUG ] 03:15:04 DEBUG "pod": "ds-cts-1" 03:15:04 DEBUG } 03:15:04 DEBUG }, 03:15:04 DEBUG ] 03:15:04 DEBUG "value": [ 03:15:04 DEBUG } 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG } 03:15:04 DEBUG "29.3" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "instance": "10.142.0.107:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "instance": "10.142.0.77:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "704366396416" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "1744.9666666666667" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974504" 03:15:04 ERROR [http_cmd]: ERROR 03:15:04 ERROR --- status code --- 03:15:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:15:04 ERROR --- http response --- 03:15:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:15:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "instance": "10.142.0.107:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "instance": "10.142.0.77:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0.05350000000001387" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "5.2371710752611325" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "48834.845452823065" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "instance": "10.142.0.107:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "instance": "10.142.0.77:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "51390988.03333333" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-cts-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-idrepo-0" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "instance": "10.142.0.107:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "instance": "10.142.0.77:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "51385455.06666666" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "1750.2666666666667" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-cts-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "ds-idrepo-1" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "ds-idrepo-0" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974504" 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "10.4620299971245" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "6239809536" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "instance": "10.142.0.107:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "instance": "10.142.0.77:9100" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0.023333333333332717" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974504" 03:15:04 INFO [http_cmd]: http status code OK 03:15:04 DEBUG --- status code --- 03:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:15:04 DEBUG --- http response --- 03:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:15:04 DEBUG ----- output ----- 03:15:04 DEBUG { 03:15:04 DEBUG "status": "success", 03:15:04 DEBUG "data": { 03:15:04 DEBUG "resultType": "vector", 03:15:04 DEBUG "result": [ 03:15:04 DEBUG { 03:15:04 DEBUG "metric": { 03:15:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG [...] 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG 03:15:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:15:04 DEBUG }, 03:15:04 DEBUG "value": [ 03:15:04 DEBUG 1712974504, 03:15:04 DEBUG "0" 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG ] 03:15:04 DEBUG } 03:15:04 DEBUG } 03:15:18 INFO 03:15:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:15:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:15:18 INFO [loop_until]: OK (rc = 0) 03:15:18 DEBUG --- stdout --- 03:15:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11922m 8578Mi am-6d96779b5d-qbg7l 10006m 8495Mi am-6d96779b5d-zn69c 11487m 8484Mi ds-cts-0 10616m 13518Mi ds-cts-1 10702m 13515Mi ds-cts-2 10828m 13518Mi ds-idrepo-0 8283m 5593Mi ds-idrepo-1 4331m 5523Mi ds-idrepo-2 4193m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 6m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5322m 4195Mi 03:15:18 DEBUG --- stderr --- 03:15:18 DEBUG 03:15:24 INFO 03:15:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:15:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:15:24 INFO [loop_until]: OK (rc = 0) 03:15:24 DEBUG --- stdout --- 03:15:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4757Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10344m 34% 10377Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13708m 45% 3094Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12228m 40% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4370Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12904m 43% 10438Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4251m 26% 6309Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10917m 68% 14434Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10985m 69% 14618Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4255m 26% 6704Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10925m 68% 14442Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8345m 52% 6385Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5394m 33% 7937Mi 13% 03:15:24 DEBUG --- stderr --- 03:15:24 DEBUG 03:16:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974563" 03:16:03 INFO [http_cmd]: http status code OK 03:16:03 DEBUG --- status code --- 03:16:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:03 DEBUG --- http response --- 03:16:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "ds-idrepo-0" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "1726.6333333333332" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0.023" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG ----- output ----- 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG { 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG "data": { 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG "result": [ 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG { 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG "metric": { 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974564" 03:16:04 DEBUG }, 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "value": [ 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG 03:16:04 DEBUG { 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG 03:16:04 DEBUG "data": { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG [...] 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG }, 03:16:04 DEBUG 03:16:04 DEBUG "result": [ 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG { 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "metric": { 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "ds-cts-1" 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG }, 03:16:04 DEBUG }, 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG "value": [ 03:16:04 DEBUG "value": [ 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "0.0011384472222222223" 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG ] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG } 03:16:04 DEBUG [...] 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG }, 03:16:04 DEBUG ] 03:16:04 DEBUG 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG } 03:16:04 DEBUG 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG } 03:16:04 DEBUG 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "1744.9333333333334" 03:16:04 DEBUG "pod": "ds-cts-1" 03:16:04 DEBUG "313675.5" 03:16:04 DEBUG ] 03:16:04 DEBUG }, 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG "value": [ 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG "29.666666666666668" 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "instance": "10.142.0.107:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "instance": "10.142.0.77:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "706787818496" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "1744.2" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974564" 03:16:04 ERROR [http_cmd]: ERROR 03:16:04 ERROR --- status code --- 03:16:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:16:04 ERROR --- http response --- 03:16:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:16:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "instance": "10.142.0.107:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "instance": "10.142.0.77:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0.18366666666665643" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "5.2489922886054226" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "instance": "10.142.0.107:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "instance": "10.142.0.77:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "51395067.333333336" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "ds-cts-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "ds-idrepo-0" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "instance": "10.142.0.107:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "instance": "10.142.0.77:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "51389709" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "ds-idrepo-0" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "1745.1666666666667" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "570.175743866365" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "ds-idrepo-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "ds-cts-1" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "6238105600" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG } 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG } 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "instance": "10.142.0.107:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "instance": "10.142.0.77:9100" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0.06399999999999934" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974564" 03:16:04 INFO [http_cmd]: http status code OK 03:16:04 DEBUG --- status code --- 03:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:16:04 DEBUG --- http response --- 03:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:16:04 DEBUG ----- output ----- 03:16:04 DEBUG { 03:16:04 DEBUG "status": "success", 03:16:04 DEBUG "data": { 03:16:04 DEBUG "resultType": "vector", 03:16:04 DEBUG "result": [ 03:16:04 DEBUG { 03:16:04 DEBUG "metric": { 03:16:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG [...] 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG 03:16:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:16:04 DEBUG }, 03:16:04 DEBUG "value": [ 03:16:04 DEBUG 1712974564, 03:16:04 DEBUG "0" 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG ] 03:16:04 DEBUG } 03:16:04 DEBUG } 03:16:18 INFO 03:16:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:16:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:16:18 INFO [loop_until]: OK (rc = 0) 03:16:18 DEBUG --- stdout --- 03:16:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12196m 8574Mi am-6d96779b5d-qbg7l 9747m 8482Mi am-6d96779b5d-zn69c 11602m 8488Mi ds-cts-0 10686m 13518Mi ds-cts-1 11055m 12794Mi ds-cts-2 10445m 13518Mi ds-idrepo-0 8180m 5594Mi ds-idrepo-1 4350m 5523Mi ds-idrepo-2 4289m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 5306m 4195Mi 03:16:18 DEBUG --- stderr --- 03:16:18 DEBUG 03:16:24 INFO 03:16:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:16:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:16:25 INFO [loop_until]: OK (rc = 0) 03:16:25 DEBUG --- stdout --- 03:16:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4732Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1868Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10272m 34% 10361Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14083m 47% 3079Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12098m 40% 10301Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 4358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12877m 43% 10423Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4375m 27% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10415m 65% 14450Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10739m 67% 13707Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 4247m 26% 6702Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10822m 68% 14449Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8267m 52% 6389Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5370m 33% 7937Mi 13% 03:16:25 DEBUG --- stderr --- 03:16:25 DEBUG 03:17:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974623" 03:17:03 INFO [http_cmd]: http status code OK 03:17:03 DEBUG --- status code --- 03:17:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:03 DEBUG --- http response --- 03:17:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-idrepo-0" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "1744.4666666666667" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:04 DEBUG }, 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-cts-1" 03:17:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:04 DEBUG }, 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0.000033333333333333335" 03:17:04 DEBUG "1751.8666666666666" 03:17:04 DEBUG ] 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-cts-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "29.566666666666666" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "1751.8" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0.023" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "314393" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "instance": "10.142.0.107:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "instance": "10.142.0.77:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "708902419456" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974624" 03:17:04 ERROR [http_cmd]: ERROR 03:17:04 ERROR --- status code --- 03:17:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:17:04 ERROR --- http response --- 03:17:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:17:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "3635.1720371381753" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "instance": "10.142.0.107:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG [...] 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "instance": "10.142.0.77:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0.10796666666666065" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "5.225069667126395" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "instance": "10.142.0.107:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "instance": "10.142.0.77:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "51055172.233333334" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-cts-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-idrepo-0" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "instance": "10.142.0.107:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "instance": "10.142.0.77:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "51049414.53333333" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-idrepo-0" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "1752.1" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "559.2572364827962" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "ds-idrepo-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "ds-cts-1" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974624" 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974624" 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974624" 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:17:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:17:04 DEBUG }, 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:17:04 DEBUG "pod": "idm-b85f9d978-dz57p" 03:17:04 DEBUG }, 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0" 03:17:04 DEBUG "0.0030030379574001597" 03:17:04 DEBUG ] 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "6237601792" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:04 INFO [http_cmd]: http status code OK 03:17:04 DEBUG --- status code --- 03:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:04 DEBUG --- http response --- 03:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:17:04 DEBUG ----- output ----- 03:17:04 DEBUG { 03:17:04 DEBUG "status": "success", 03:17:04 DEBUG "data": { 03:17:04 DEBUG "resultType": "vector", 03:17:04 DEBUG "result": [ 03:17:04 DEBUG { 03:17:04 DEBUG "metric": { 03:17:04 DEBUG "instance": "10.142.0.107:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG [...] 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG 03:17:04 DEBUG "instance": "10.142.0.77:9100" 03:17:04 DEBUG }, 03:17:04 DEBUG "value": [ 03:17:04 DEBUG 1712974624, 03:17:04 DEBUG "0.03666666666666671" 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG ] 03:17:04 DEBUG } 03:17:04 DEBUG } 03:17:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974625" 03:17:05 INFO [http_cmd]: http status code OK 03:17:05 DEBUG --- status code --- 03:17:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:17:05 DEBUG --- http response --- 03:17:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:17:05 DEBUG ----- output ----- 03:17:05 DEBUG { 03:17:05 DEBUG "status": "success", 03:17:05 DEBUG "data": { 03:17:05 DEBUG "resultType": "vector", 03:17:05 DEBUG "result": [ 03:17:05 DEBUG { 03:17:05 DEBUG "metric": { 03:17:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:17:05 DEBUG }, 03:17:05 DEBUG "value": [ 03:17:05 DEBUG 03:17:05 DEBUG 03:17:05 DEBUG 03:17:05 DEBUG [...] 03:17:05 DEBUG 03:17:05 DEBUG 03:17:05 DEBUG 03:17:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:17:05 DEBUG }, 03:17:05 DEBUG "value": [ 03:17:05 DEBUG 1712974625, 03:17:05 DEBUG "0" 03:17:05 DEBUG ] 03:17:05 DEBUG } 03:17:05 DEBUG ] 03:17:05 DEBUG } 03:17:05 DEBUG } 03:17:18 INFO 03:17:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:17:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:17:18 INFO [loop_until]: OK (rc = 0) 03:17:18 DEBUG --- stdout --- 03:17:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 11890m 8587Mi am-6d96779b5d-qbg7l 9990m 8487Mi am-6d96779b5d-zn69c 11951m 8487Mi ds-cts-0 10857m 13516Mi ds-cts-1 10795m 13596Mi ds-cts-2 10637m 13519Mi ds-idrepo-0 8141m 5593Mi ds-idrepo-1 4323m 5523Mi ds-idrepo-2 4103m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 1m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5183m 4195Mi 03:17:18 DEBUG --- stderr --- 03:17:18 DEBUG 03:17:25 INFO 03:17:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:17:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:17:25 INFO [loop_until]: OK (rc = 0) 03:17:25 DEBUG --- stdout --- 03:17:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4738Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1867Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10351m 34% 10365Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14088m 47% 3094Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12147m 40% 10302Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 128m 0% 4360Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12724m 42% 10438Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4357m 27% 6308Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10756m 67% 14439Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11231m 70% 14438Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4238m 26% 6703Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10709m 67% 14452Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8258m 51% 6393Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5217m 32% 7936Mi 13% 03:17:25 DEBUG --- stderr --- 03:17:25 DEBUG 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-idrepo-0" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "1744.1666666666667" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "1738.3333333333333" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "1741.8666666666666" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-cts-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0.000604775" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-cts-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "28.866666666666667" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0.023" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "314326" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "instance": "10.142.0.107:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "instance": "10.142.0.77:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "711426429952" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974684" 03:18:04 ERROR [http_cmd]: ERROR 03:18:04 ERROR --- status code --- 03:18:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:18:04 ERROR --- http response --- 03:18:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:18:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "instance": "10.142.0.107:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "instance": "10.142.0.77:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0.1778999999999925" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "248155598.66405365" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "5.352667264230582" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "instance": "10.142.0.107:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "instance": "10.142.0.77:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "51450798.3" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-cts-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-idrepo-0" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "instance": "10.142.0.107:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "instance": "10.142.0.77:9100" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "51447115.766666666" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "1741.8333333333333" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-idrepo-0" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "8371197.88015126" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974684" 03:18:04 INFO [http_cmd]: http status code OK 03:18:04 DEBUG --- status code --- 03:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:04 DEBUG --- http response --- 03:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:18:04 DEBUG ----- output ----- 03:18:04 DEBUG { 03:18:04 DEBUG "status": "success", 03:18:04 DEBUG "data": { 03:18:04 DEBUG "resultType": "vector", 03:18:04 DEBUG "result": [ 03:18:04 DEBUG { 03:18:04 DEBUG "metric": { 03:18:04 DEBUG "pod": "ds-idrepo-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG [...] 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG 03:18:04 DEBUG "pod": "ds-cts-1" 03:18:04 DEBUG }, 03:18:04 DEBUG "value": [ 03:18:04 DEBUG 1712974684, 03:18:04 DEBUG "0" 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG ] 03:18:04 DEBUG } 03:18:04 DEBUG } 03:18:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974685" 03:18:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974685" 03:18:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974685" 03:18:05 INFO [http_cmd]: http status code OK 03:18:05 DEBUG --- status code --- 03:18:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:05 DEBUG --- http response --- 03:18:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:18:05 DEBUG ----- output ----- 03:18:05 DEBUG { 03:18:05 DEBUG "status": "success", 03:18:05 DEBUG "data": { 03:18:05 DEBUG "resultType": "vector", 03:18:05 DEBUG "result": [ 03:18:05 DEBUG { 03:18:05 DEBUG "metric": { 03:18:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG [...] 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 1712974685, 03:18:05 DEBUG "0" 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG } 03:18:05 INFO [http_cmd]: http status code OK 03:18:05 DEBUG --- status code --- 03:18:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:05 DEBUG --- http response --- 03:18:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:05 DEBUG ----- output ----- 03:18:05 DEBUG { 03:18:05 DEBUG "status": "success", 03:18:05 DEBUG "data": { 03:18:05 DEBUG "resultType": "vector", 03:18:05 DEBUG "result": [ 03:18:05 DEBUG { 03:18:05 DEBUG "metric": { 03:18:05 DEBUG "instance": "10.142.0.107:9100" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG [...] 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG "instance": "10.142.0.77:9100" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 1712974685, 03:18:05 DEBUG "0.05499999999999901" 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG } 03:18:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974685" 03:18:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974685" 03:18:05 INFO [http_cmd]: http status code OK 03:18:05 DEBUG --- status code --- 03:18:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:05 DEBUG --- http response --- 03:18:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:18:05 DEBUG ----- output ----- 03:18:05 DEBUG { 03:18:05 DEBUG "status": "success", 03:18:05 DEBUG "data": { 03:18:05 DEBUG "resultType": "vector", 03:18:05 DEBUG "result": [ 03:18:05 DEBUG { 03:18:05 DEBUG "metric": { 03:18:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG [...] 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 1712974685, 03:18:05 DEBUG "6237294592" 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG } 03:18:05 INFO [http_cmd]: http status code OK 03:18:05 DEBUG --- status code --- 03:18:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:05 DEBUG --- http response --- 03:18:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:18:05 DEBUG ----- output ----- 03:18:05 DEBUG { 03:18:05 DEBUG "status": "success", 03:18:05 DEBUG "data": { 03:18:05 DEBUG "resultType": "vector", 03:18:05 DEBUG "result": [ 03:18:05 DEBUG { 03:18:05 DEBUG "metric": { 03:18:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG [...] 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 1712974685, 03:18:05 DEBUG "0" 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG } 03:18:05 INFO [http_cmd]: http status code OK 03:18:05 DEBUG --- status code --- 03:18:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:18:05 DEBUG --- http response --- 03:18:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:18:05 DEBUG ----- output ----- 03:18:05 DEBUG { 03:18:05 DEBUG "status": "success", 03:18:05 DEBUG "data": { 03:18:05 DEBUG "resultType": "vector", 03:18:05 DEBUG "result": [ 03:18:05 DEBUG { 03:18:05 DEBUG "metric": { 03:18:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG [...] 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG 03:18:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:18:05 DEBUG }, 03:18:05 DEBUG "value": [ 03:18:05 DEBUG 1712974685, 03:18:05 DEBUG "12.13409324022171" 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG ] 03:18:05 DEBUG } 03:18:05 DEBUG } 03:18:18 INFO 03:18:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:18:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:18:18 INFO [loop_until]: OK (rc = 0) 03:18:18 DEBUG --- stdout --- 03:18:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12346m 8577Mi am-6d96779b5d-qbg7l 9868m 8486Mi am-6d96779b5d-zn69c 11699m 8489Mi ds-cts-0 11073m 13612Mi ds-cts-1 11034m 13516Mi ds-cts-2 10417m 13519Mi ds-idrepo-0 8303m 5594Mi ds-idrepo-1 4306m 5523Mi ds-idrepo-2 4169m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5184m 4196Mi 03:18:18 DEBUG --- stderr --- 03:18:18 DEBUG 03:18:25 INFO 03:18:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:18:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:18:25 INFO [loop_until]: OK (rc = 0) 03:18:25 DEBUG --- stdout --- 03:18:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4746Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1872Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10445m 34% 10358Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14094m 47% 3104Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12159m 40% 10301Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4345Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12928m 43% 10429Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4416m 27% 6308Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10818m 68% 14446Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10892m 68% 14410Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4097m 25% 6707Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10948m 68% 14613Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8268m 52% 6392Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5379m 33% 7935Mi 13% 03:18:25 DEBUG --- stderr --- 03:18:25 DEBUG 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-idrepo-0" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "1752.7" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "1749.9333333333334" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-cts-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "27" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "1756.3999999999999" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "instance": "10.142.0.107:9100" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "instance": "10.142.0.77:9100" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "713996000256" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-cts-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0.000049661111111111114" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:04 DEBUG }, 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:04 DEBUG }, 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "314776.5" 03:19:04 DEBUG "0.023" 03:19:04 DEBUG ] 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974744" 03:19:04 ERROR [http_cmd]: ERROR 03:19:04 ERROR --- status code --- 03:19:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:19:04 ERROR --- http response --- 03:19:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:19:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974744" 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "5.261228579376704" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "instance": "10.142.0.107:9100" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "instance": "10.142.0.77:9100" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0.213033333333334" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "instance": "10.142.0.107:9100" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "instance": "10.142.0.77:9100" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "51795170.03333333" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "4591.268358577875" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-cts-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974744" 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-idrepo-0" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:04 DEBUG { 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG { 03:19:04 DEBUG "data": { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "result": [ 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG { 03:19:04 DEBUG "result": [ 03:19:04 DEBUG "metric": { 03:19:04 DEBUG { 03:19:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:04 DEBUG "metric": { 03:19:04 DEBUG }, 03:19:04 DEBUG "instance": "10.142.0.107:9100" 03:19:04 DEBUG "value": [ 03:19:04 DEBUG }, 03:19:04 DEBUG 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:04 DEBUG 03:19:04 DEBUG }, 03:19:04 DEBUG "instance": "10.142.0.77:9100" 03:19:04 DEBUG "value": [ 03:19:04 DEBUG }, 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG "1756.6" 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG ] 03:19:04 DEBUG "51789114.63333333" 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "2943.120742678125" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-idrepo-0" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974744" 03:19:04 INFO [http_cmd]: http status code OK 03:19:04 DEBUG --- status code --- 03:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:04 DEBUG --- http response --- 03:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:19:04 DEBUG ----- output ----- 03:19:04 DEBUG { 03:19:04 DEBUG "status": "success", 03:19:04 DEBUG "data": { 03:19:04 DEBUG "resultType": "vector", 03:19:04 DEBUG "result": [ 03:19:04 DEBUG { 03:19:04 DEBUG "metric": { 03:19:04 DEBUG "pod": "ds-idrepo-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG [...] 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG 03:19:04 DEBUG "pod": "ds-cts-1" 03:19:04 DEBUG }, 03:19:04 DEBUG "value": [ 03:19:04 DEBUG 1712974744, 03:19:04 DEBUG "0" 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG ] 03:19:04 DEBUG } 03:19:04 DEBUG } 03:19:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974745" 03:19:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974745" 03:19:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974745" 03:19:05 INFO [http_cmd]: http status code OK 03:19:05 DEBUG --- status code --- 03:19:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:05 DEBUG --- http response --- 03:19:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:19:05 DEBUG ----- output ----- 03:19:05 DEBUG { 03:19:05 DEBUG "status": "success", 03:19:05 DEBUG "data": { 03:19:05 DEBUG "resultType": "vector", 03:19:05 DEBUG "result": [ 03:19:05 DEBUG { 03:19:05 DEBUG "metric": { 03:19:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG [...] 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG "pod": "ds-idrepo-1" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 1712974745, 03:19:05 DEBUG "4.306734737619686" 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG } 03:19:05 INFO [http_cmd]: http status code OK 03:19:05 DEBUG --- status code --- 03:19:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:05 DEBUG --- http response --- 03:19:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:19:05 DEBUG ----- output ----- 03:19:05 DEBUG { 03:19:05 DEBUG "status": "success", 03:19:05 DEBUG "data": { 03:19:05 DEBUG "resultType": "vector", 03:19:05 DEBUG "result": [ 03:19:05 DEBUG { 03:19:05 DEBUG "metric": { 03:19:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG [...] 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 1712974745, 03:19:05 DEBUG "0" 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG } 03:19:05 INFO [http_cmd]: http status code OK 03:19:05 DEBUG --- status code --- 03:19:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:05 DEBUG --- http response --- 03:19:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:05 DEBUG ----- output ----- 03:19:05 DEBUG { 03:19:05 DEBUG "status": "success", 03:19:05 DEBUG "data": { 03:19:05 DEBUG "resultType": "vector", 03:19:05 DEBUG "result": [ 03:19:05 DEBUG { 03:19:05 DEBUG "metric": { 03:19:05 DEBUG "instance": "10.142.0.107:9100" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG [...] 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG "instance": "10.142.0.77:9100" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 1712974745, 03:19:05 DEBUG "0.0623333333333304" 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG } 03:19:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974745" 03:19:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974745" 03:19:05 INFO [http_cmd]: http status code OK 03:19:05 DEBUG --- status code --- 03:19:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:05 DEBUG --- http response --- 03:19:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:19:05 DEBUG ----- output ----- 03:19:05 DEBUG { 03:19:05 DEBUG "status": "success", 03:19:05 DEBUG "data": { 03:19:05 DEBUG "resultType": "vector", 03:19:05 DEBUG "result": [ 03:19:05 DEBUG { 03:19:05 DEBUG "metric": { 03:19:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG [...] 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 1712974745, 03:19:05 DEBUG "6237474816" 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG } 03:19:05 INFO [http_cmd]: http status code OK 03:19:05 DEBUG --- status code --- 03:19:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:19:05 DEBUG --- http response --- 03:19:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:19:05 DEBUG ----- output ----- 03:19:05 DEBUG { 03:19:05 DEBUG "status": "success", 03:19:05 DEBUG "data": { 03:19:05 DEBUG "resultType": "vector", 03:19:05 DEBUG "result": [ 03:19:05 DEBUG { 03:19:05 DEBUG "metric": { 03:19:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG [...] 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG 03:19:05 DEBUG "pod": "ds-idrepo-1" 03:19:05 DEBUG }, 03:19:05 DEBUG "value": [ 03:19:05 DEBUG 1712974745, 03:19:05 DEBUG "0" 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG ] 03:19:05 DEBUG } 03:19:05 DEBUG } 03:19:18 INFO 03:19:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:19:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:19:18 INFO [loop_until]: OK (rc = 0) 03:19:18 DEBUG --- stdout --- 03:19:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12132m 8575Mi am-6d96779b5d-qbg7l 9875m 8482Mi am-6d96779b5d-zn69c 12122m 8487Mi ds-cts-0 10654m 13519Mi ds-cts-1 11416m 13684Mi ds-cts-2 10145m 13651Mi ds-idrepo-0 7947m 5594Mi ds-idrepo-1 4318m 5523Mi ds-idrepo-2 4263m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1121Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 7m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5201m 4195Mi 03:19:18 DEBUG --- stderr --- 03:19:18 DEBUG 03:19:25 INFO 03:19:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:19:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:19:25 INFO [loop_until]: OK (rc = 0) 03:19:25 DEBUG --- stdout --- 03:19:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4740Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1874Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9881m 33% 10358Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13643m 45% 3097Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12128m 40% 10293Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4340Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12829m 42% 10424Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4425m 27% 6311Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10413m 65% 14441Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11376m 71% 14408Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4222m 26% 6707Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10815m 68% 14465Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8261m 51% 6394Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5399m 33% 7936Mi 13% 03:19:25 DEBUG --- stderr --- 03:19:25 DEBUG 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-idrepo-0" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "1736.3" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "instance": "10.142.0.107:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "instance": "10.142.0.77:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "716181330944" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "1738.5" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "1749.3" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-cts-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "29.466666666666665" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0.023" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-cts-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0.00016666666666666674" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974804" 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 ERROR [http_cmd]: ERROR 03:20:04 DEBUG --- status code --- 03:20:04 ERROR --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 ERROR --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:20:04 DEBUG ----- output ----- 03:20:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "314537" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "instance": "10.142.0.107:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "instance": "10.142.0.77:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0.16880000000000167" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "6839.309025774916" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "5.273561236481814" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "instance": "10.142.0.107:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "instance": "10.142.0.77:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "51411041.2" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-idrepo-0" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "idm-b85f9d978-b559t" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "497.4042927836303" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-cts-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "1735.3" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "instance": "10.142.0.107:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "instance": "10.142.0.77:9100" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "51405511.8" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-idrepo-0" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974804" 03:20:04 INFO [http_cmd]: http status code OK 03:20:04 DEBUG --- status code --- 03:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:04 DEBUG --- http response --- 03:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:20:04 DEBUG ----- output ----- 03:20:04 DEBUG { 03:20:04 DEBUG "status": "success", 03:20:04 DEBUG "data": { 03:20:04 DEBUG "resultType": "vector", 03:20:04 DEBUG "result": [ 03:20:04 DEBUG { 03:20:04 DEBUG "metric": { 03:20:04 DEBUG "pod": "ds-idrepo-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG [...] 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG 03:20:04 DEBUG "pod": "ds-cts-1" 03:20:04 DEBUG }, 03:20:04 DEBUG "value": [ 03:20:04 DEBUG 1712974804, 03:20:04 DEBUG "0" 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG ] 03:20:04 DEBUG } 03:20:04 DEBUG } 03:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974805" 03:20:05 INFO [http_cmd]: http status code OK 03:20:05 DEBUG --- status code --- 03:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:05 DEBUG --- http response --- 03:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:20:05 DEBUG ----- output ----- 03:20:05 DEBUG { 03:20:05 DEBUG "status": "success", 03:20:05 DEBUG "data": { 03:20:05 DEBUG "resultType": "vector", 03:20:05 DEBUG "result": [ 03:20:05 DEBUG { 03:20:05 DEBUG "metric": { 03:20:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG [...] 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG "pod": "ds-cts-1" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 1712974805, 03:20:05 DEBUG "11.09400670237133" 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG } 03:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974805" 03:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974805" 03:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974805" 03:20:05 INFO [http_cmd]: http status code OK 03:20:05 DEBUG --- status code --- 03:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:05 DEBUG --- http response --- 03:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:05 DEBUG ----- output ----- 03:20:05 DEBUG { 03:20:05 DEBUG "status": "success", 03:20:05 DEBUG "data": { 03:20:05 DEBUG "resultType": "vector", 03:20:05 DEBUG "result": [ 03:20:05 DEBUG { 03:20:05 DEBUG "metric": { 03:20:05 DEBUG "instance": "10.142.0.107:9100" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG [...] 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG "instance": "10.142.0.77:9100" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 1712974805, 03:20:05 DEBUG "0.050666666666666756" 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG } 03:20:05 INFO [http_cmd]: http status code OK 03:20:05 DEBUG --- status code --- 03:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:05 DEBUG --- http response --- 03:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:20:05 DEBUG ----- output ----- 03:20:05 DEBUG { 03:20:05 DEBUG "status": "success", 03:20:05 DEBUG "data": { 03:20:05 DEBUG "resultType": "vector", 03:20:05 DEBUG "result": [ 03:20:05 DEBUG { 03:20:05 DEBUG "metric": { 03:20:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG [...] 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 1712974805, 03:20:05 DEBUG "0" 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG } 03:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974805" 03:20:05 INFO [http_cmd]: http status code OK 03:20:05 DEBUG --- status code --- 03:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:05 DEBUG --- http response --- 03:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:05 DEBUG ----- output ----- 03:20:05 DEBUG { 03:20:05 DEBUG "status": "success", 03:20:05 DEBUG "data": { 03:20:05 DEBUG "resultType": "vector", 03:20:05 DEBUG "result": [ 03:20:05 DEBUG { 03:20:05 DEBUG "metric": { 03:20:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG [...] 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG "pod": "ds-cts-1" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 1712974805, 03:20:05 DEBUG "24690510.5238483" 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG } 03:20:05 INFO [http_cmd]: http status code OK 03:20:05 DEBUG --- status code --- 03:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:20:05 DEBUG --- http response --- 03:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:20:05 DEBUG ----- output ----- 03:20:05 DEBUG { 03:20:05 DEBUG "status": "success", 03:20:05 DEBUG "data": { 03:20:05 DEBUG "resultType": "vector", 03:20:05 DEBUG "result": [ 03:20:05 DEBUG { 03:20:05 DEBUG "metric": { 03:20:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG [...] 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG 03:20:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:20:05 DEBUG }, 03:20:05 DEBUG "value": [ 03:20:05 DEBUG 1712974805, 03:20:05 DEBUG "6237622272" 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG ] 03:20:05 DEBUG } 03:20:05 DEBUG } 03:20:18 INFO 03:20:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:20:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:20:18 INFO [loop_until]: OK (rc = 0) 03:20:18 DEBUG --- stdout --- 03:20:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12261m 8576Mi am-6d96779b5d-qbg7l 9912m 8482Mi am-6d96779b5d-zn69c 11721m 8482Mi ds-cts-0 10702m 13517Mi ds-cts-1 10929m 13551Mi ds-cts-2 10234m 13521Mi ds-idrepo-0 8070m 5594Mi ds-idrepo-1 4218m 5523Mi ds-idrepo-2 4131m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5183m 4195Mi 03:20:18 DEBUG --- stderr --- 03:20:18 DEBUG 03:20:25 INFO 03:20:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:20:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:20:25 INFO [loop_until]: OK (rc = 0) 03:20:25 DEBUG --- stdout --- 03:20:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4736Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10290m 34% 10366Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13975m 46% 3101Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11930m 39% 10288Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4346Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12708m 42% 10439Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4262m 26% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10742m 67% 14459Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 10828m 68% 14500Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4248m 26% 6707Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10990m 69% 14565Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8316m 52% 6393Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5343m 33% 7935Mi 13% 03:20:25 DEBUG --- stderr --- 03:20:25 DEBUG 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "ds-idrepo-0" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "instance": "10.142.0.107:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "instance": "10.142.0.77:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "718637038592" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG { 03:21:04 DEBUG --- http response --- 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:21:04 DEBUG "data": { 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG { 03:21:04 DEBUG "result": [ 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG { 03:21:04 DEBUG "data": { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG "result": [ 03:21:04 DEBUG }, 03:21:04 DEBUG { 03:21:04 DEBUG "value": [ 03:21:04 DEBUG "metric": { 03:21:04 DEBUG 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG 03:21:04 DEBUG }, 03:21:04 DEBUG 03:21:04 DEBUG "value": [ 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG [...] 03:21:04 DEBUG "pod": "ds-cts-1" 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG 03:21:04 DEBUG }, 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG 1712974864, 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG "pod": "ds-cts-1" 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG "0.0002" 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG }, 03:21:04 DEBUG { 03:21:04 DEBUG ] 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG "value": [ 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG } 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "data": { 03:21:04 DEBUG ] 03:21:04 DEBUG "29.266666666666666" 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG "result": [ 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 DEBUG { 03:21:04 DEBUG { 03:21:04 DEBUG ] 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "metric": { 03:21:04 DEBUG } 03:21:04 DEBUG "data": { 03:21:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:04 DEBUG } 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG }, 03:21:04 DEBUG "result": [ 03:21:04 DEBUG "value": [ 03:21:04 DEBUG { 03:21:04 DEBUG 03:21:04 DEBUG "metric": { 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:04 DEBUG 03:21:04 DEBUG }, 03:21:04 DEBUG [...] 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG [...] 03:21:04 DEBUG }, 03:21:04 DEBUG 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG 03:21:04 DEBUG "1751.9333333333334" 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG ] 03:21:04 DEBUG }, 03:21:04 DEBUG } 03:21:04 DEBUG "value": [ 03:21:04 DEBUG ] 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG } 03:21:04 DEBUG "1747.7333333333333" 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "1750.3666666666666" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0.023" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974864" 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 ERROR [http_cmd]: ERROR 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 ERROR --- status code --- 03:21:04 DEBUG --- status code --- 03:21:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 ERROR --- http response --- 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:21:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "314924.5" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "instance": "10.142.0.107:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "instance": "10.142.0.77:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0.19986666666666936" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "301671630.2983521" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "5.235612088680458" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974864" 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "instance": "10.142.0.107:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "instance": "10.142.0.77:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "51372850.96666667" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "ds-cts-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "ds-idrepo-0" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "6151766.687921601" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974864" 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "1752.3" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "instance": "10.142.0.107:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "instance": "10.142.0.77:9100" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "51367225.46666667" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "ds-idrepo-0" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974864" 03:21:04 INFO [http_cmd]: http status code OK 03:21:04 DEBUG --- status code --- 03:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:04 DEBUG --- http response --- 03:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:21:04 DEBUG ----- output ----- 03:21:04 DEBUG { 03:21:04 DEBUG "status": "success", 03:21:04 DEBUG "data": { 03:21:04 DEBUG "resultType": "vector", 03:21:04 DEBUG "result": [ 03:21:04 DEBUG { 03:21:04 DEBUG "metric": { 03:21:04 DEBUG "pod": "ds-idrepo-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG [...] 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG 03:21:04 DEBUG "pod": "ds-cts-1" 03:21:04 DEBUG }, 03:21:04 DEBUG "value": [ 03:21:04 DEBUG 1712974864, 03:21:04 DEBUG "0" 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG ] 03:21:04 DEBUG } 03:21:04 DEBUG } 03:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974865" 03:21:05 INFO [http_cmd]: http status code OK 03:21:05 DEBUG --- status code --- 03:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:05 DEBUG --- http response --- 03:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:05 DEBUG ----- output ----- 03:21:05 DEBUG { 03:21:05 DEBUG "status": "success", 03:21:05 DEBUG "data": { 03:21:05 DEBUG "resultType": "vector", 03:21:05 DEBUG "result": [ 03:21:05 DEBUG { 03:21:05 DEBUG "metric": { 03:21:05 DEBUG "instance": "10.142.0.107:9100" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG [...] 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG "instance": "10.142.0.77:9100" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 1712974865, 03:21:05 DEBUG "0.05766666666666395" 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG } 03:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974865" 03:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974865" 03:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974865" 03:21:05 INFO [http_cmd]: http status code OK 03:21:05 DEBUG --- status code --- 03:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:05 DEBUG --- http response --- 03:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:21:05 DEBUG ----- output ----- 03:21:05 DEBUG { 03:21:05 DEBUG "status": "success", 03:21:05 DEBUG "data": { 03:21:05 DEBUG "resultType": "vector", 03:21:05 DEBUG "result": [ 03:21:05 DEBUG { 03:21:05 DEBUG "metric": { 03:21:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG [...] 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 1712974865, 03:21:05 DEBUG "6239141888" 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG } 03:21:05 INFO [http_cmd]: http status code OK 03:21:05 DEBUG --- status code --- 03:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:05 DEBUG --- http response --- 03:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:21:05 DEBUG ----- output ----- 03:21:05 DEBUG { 03:21:05 DEBUG "status": "success", 03:21:05 DEBUG "data": { 03:21:05 DEBUG "resultType": "vector", 03:21:05 DEBUG "result": [ 03:21:05 DEBUG { 03:21:05 DEBUG "metric": { 03:21:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG [...] 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 1712974865, 03:21:05 DEBUG "12.046863677552004" 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG } 03:21:05 INFO [http_cmd]: http status code OK 03:21:05 DEBUG --- status code --- 03:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:05 DEBUG --- http response --- 03:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:21:05 DEBUG ----- output ----- 03:21:05 DEBUG { 03:21:05 DEBUG "status": "success", 03:21:05 DEBUG "data": { 03:21:05 DEBUG "resultType": "vector", 03:21:05 DEBUG "result": [ 03:21:05 DEBUG { 03:21:05 DEBUG "metric": { 03:21:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG [...] 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 1712974865, 03:21:05 DEBUG "0" 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG } 03:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974865" 03:21:05 INFO [http_cmd]: http status code OK 03:21:05 DEBUG --- status code --- 03:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:21:05 DEBUG --- http response --- 03:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:21:05 DEBUG ----- output ----- 03:21:05 DEBUG { 03:21:05 DEBUG "status": "success", 03:21:05 DEBUG "data": { 03:21:05 DEBUG "resultType": "vector", 03:21:05 DEBUG "result": [ 03:21:05 DEBUG { 03:21:05 DEBUG "metric": { 03:21:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG [...] 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG 03:21:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:21:05 DEBUG }, 03:21:05 DEBUG "value": [ 03:21:05 DEBUG 1712974865, 03:21:05 DEBUG "0" 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG ] 03:21:05 DEBUG } 03:21:05 DEBUG } 03:21:18 INFO 03:21:18 INFO [loop_until]: kubectl --namespace=xlou top pods 03:21:18 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:21:18 INFO [loop_until]: OK (rc = 0) 03:21:18 DEBUG --- stdout --- 03:21:18 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 12144m 8573Mi am-6d96779b5d-qbg7l 9733m 8484Mi am-6d96779b5d-zn69c 11720m 8484Mi ds-cts-0 10831m 13517Mi ds-cts-1 10826m 13546Mi ds-cts-2 10293m 13524Mi ds-idrepo-0 8207m 5594Mi ds-idrepo-1 4299m 5523Mi ds-idrepo-2 4145m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 5m 1121Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 2m 67Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 5171m 4196Mi 03:21:18 DEBUG --- stderr --- 03:21:18 DEBUG 03:21:25 INFO 03:21:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:21:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:21:25 INFO [loop_until]: OK (rc = 0) 03:21:25 DEBUG --- stdout --- 03:21:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4741Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 1861Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10214m 34% 10361Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14036m 47% 3099Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12216m 40% 10291Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4348Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12901m 43% 10426Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 4374m 27% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 10552m 66% 14443Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 11291m 71% 14435Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 4228m 26% 6707Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 10911m 68% 14459Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 8257m 51% 6391Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5368m 33% 7937Mi 13% 03:21:25 DEBUG --- stderr --- 03:21:25 DEBUG 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-idrepo-0" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "instance": "10.142.0.107:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "instance": "10.142.0.77:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "720893469696" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "1739.3666666666666" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-cts-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "29.3" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "1739.8333333333333" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "1752.7666666666667" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-cts-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0.0002816388888888889" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0.023" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974924" 03:22:04 ERROR [http_cmd]: ERROR 03:22:04 ERROR --- status code --- 03:22:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:22:04 ERROR --- http response --- 03:22:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:22:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "314834.5" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "instance": "10.142.0.107:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "instance": "10.142.0.77:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0.17436666666666648" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG { 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:22:04 DEBUG { 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG "metric": { 03:22:04 DEBUG { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG }, 03:22:04 DEBUG "data": { 03:22:04 DEBUG "value": [ 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG 03:22:04 DEBUG "result": [ 03:22:04 DEBUG 03:22:04 DEBUG { 03:22:04 DEBUG 03:22:04 DEBUG "metric": { 03:22:04 DEBUG [...] 03:22:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:22:04 DEBUG 03:22:04 DEBUG }, 03:22:04 DEBUG 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-idrepo-0" 03:22:04 DEBUG 03:22:04 DEBUG }, 03:22:04 DEBUG 03:22:04 DEBUG "value": [ 03:22:04 DEBUG [...] 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG 03:22:04 DEBUG "0" 03:22:04 DEBUG 03:22:04 DEBUG ] 03:22:04 DEBUG 03:22:04 DEBUG } 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG ] 03:22:04 DEBUG }, 03:22:04 DEBUG } 03:22:04 DEBUG "value": [ 03:22:04 DEBUG } 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "5.247620892907562" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "instance": "10.142.0.107:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "instance": "10.142.0.77:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "51499173.9" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-cts-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "1739.5333333333333" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "instance": "10.142.0.107:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "instance": "10.142.0.77:9100" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "51493384.36666667" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-idrepo-0" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974924" 03:22:04 INFO [http_cmd]: http status code OK 03:22:04 DEBUG --- status code --- 03:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:04 DEBUG --- http response --- 03:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:22:04 DEBUG ----- output ----- 03:22:04 DEBUG { 03:22:04 DEBUG "status": "success", 03:22:04 DEBUG "data": { 03:22:04 DEBUG "resultType": "vector", 03:22:04 DEBUG "result": [ 03:22:04 DEBUG { 03:22:04 DEBUG "metric": { 03:22:04 DEBUG "pod": "ds-idrepo-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG [...] 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG 03:22:04 DEBUG "pod": "ds-cts-1" 03:22:04 DEBUG }, 03:22:04 DEBUG "value": [ 03:22:04 DEBUG 1712974924, 03:22:04 DEBUG "0" 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG ] 03:22:04 DEBUG } 03:22:04 DEBUG } 03:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974925" 03:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974925" 03:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974925" 03:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974925" 03:22:05 INFO [http_cmd]: http status code OK 03:22:05 DEBUG --- status code --- 03:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:05 DEBUG --- http response --- 03:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:05 DEBUG ----- output ----- 03:22:05 DEBUG { 03:22:05 DEBUG "status": "success", 03:22:05 DEBUG "data": { 03:22:05 DEBUG "resultType": "vector", 03:22:05 DEBUG "result": [ 03:22:05 DEBUG { 03:22:05 DEBUG "metric": { 03:22:05 DEBUG "instance": "10.142.0.107:9100" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG [...] 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG "instance": "10.142.0.77:9100" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 1712974925, 03:22:05 DEBUG "0.05333333333333504" 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG } 03:22:05 INFO [http_cmd]: http status code OK 03:22:05 DEBUG --- status code --- 03:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:05 DEBUG --- http response --- 03:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:22:05 DEBUG ----- output ----- 03:22:05 DEBUG { 03:22:05 DEBUG "status": "success", 03:22:05 DEBUG "data": { 03:22:05 DEBUG "resultType": "vector", 03:22:05 DEBUG "result": [ 03:22:05 DEBUG { 03:22:05 DEBUG "metric": { 03:22:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG [...] 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG "pod": "ds-idrepo-1" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 1712974925, 03:22:05 DEBUG "0" 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG } 03:22:05 INFO [http_cmd]: http status code OK 03:22:05 DEBUG --- status code --- 03:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:05 DEBUG --- http response --- 03:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:05 DEBUG ----- output ----- 03:22:05 DEBUG { 03:22:05 DEBUG "status": "success", 03:22:05 DEBUG "data": { 03:22:05 DEBUG "resultType": "vector", 03:22:05 DEBUG "result": [ 03:22:05 DEBUG { 03:22:05 DEBUG "metric": { 03:22:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG [...] 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 1712974925, 03:22:05 DEBUG "6239432704" 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG } 03:22:05 INFO [http_cmd]: http status code OK 03:22:05 DEBUG --- status code --- 03:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:05 DEBUG --- http response --- 03:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:22:05 DEBUG ----- output ----- 03:22:05 DEBUG { 03:22:05 DEBUG "status": "success", 03:22:05 DEBUG "data": { 03:22:05 DEBUG "resultType": "vector", 03:22:05 DEBUG "result": [ 03:22:05 DEBUG { 03:22:05 DEBUG "metric": { 03:22:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG [...] 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 1712974925, 03:22:05 DEBUG "0" 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG } 03:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974925" 03:22:05 INFO [http_cmd]: http status code OK 03:22:05 DEBUG --- status code --- 03:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:22:05 DEBUG --- http response --- 03:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:22:05 DEBUG ----- output ----- 03:22:05 DEBUG { 03:22:05 DEBUG "status": "success", 03:22:05 DEBUG "data": { 03:22:05 DEBUG "resultType": "vector", 03:22:05 DEBUG "result": [ 03:22:05 DEBUG { 03:22:05 DEBUG "metric": { 03:22:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG [...] 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG 03:22:05 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:22:05 DEBUG }, 03:22:05 DEBUG "value": [ 03:22:05 DEBUG 1712974925, 03:22:05 DEBUG "0" 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG ] 03:22:05 DEBUG } 03:22:05 DEBUG } 03:22:19 INFO 03:22:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:22:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:22:19 INFO [loop_until]: OK (rc = 0) 03:22:19 DEBUG --- stdout --- 03:22:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 3050m 8570Mi am-6d96779b5d-qbg7l 1686m 8486Mi am-6d96779b5d-zn69c 4005m 8486Mi ds-cts-0 4866m 13054Mi ds-cts-1 4790m 13538Mi ds-cts-2 4807m 13510Mi ds-idrepo-0 919m 5594Mi ds-idrepo-1 916m 5523Mi ds-idrepo-2 1102m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 4m 68Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 806m 3466Mi 03:22:19 DEBUG --- stderr --- 03:22:19 DEBUG 03:22:25 INFO 03:22:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:22:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:22:25 INFO [loop_until]: OK (rc = 0) 03:22:25 DEBUG --- stdout --- 03:22:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4730Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1868Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2208m 7% 10402Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 3086Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2806m 9% 10326Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4344Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2176m 7% 10451Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 444m 2% 6310Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4928m 31% 14456Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 5110m 32% 14415Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6709Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 4900m 30% 14026Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 1003m 6% 6393Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 71m 0% 7215Mi 12% 03:22:25 DEBUG --- stderr --- 03:22:25 DEBUG 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "instance": "10.142.0.107:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "instance": "10.142.0.77:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "721795421184" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-idrepo-0" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "1738.7" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-cts-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0.0001738138888888889" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 ERROR [http_cmd]: ERROR 03:23:04 ERROR --- status code --- 03:23:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:23:04 ERROR --- http response --- 03:23:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:23:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-cts-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "28.2" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0.023" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "241217.5" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "instance": "10.142.0.107:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "instance": "10.142.0.77:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0.09163333333332606" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-idrepo-0" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "262315876.9973222" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0.026707421053054674" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "instance": "10.142.0.107:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "instance": "10.142.0.77:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "11484887.066666666" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-cts-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "2067827.3254274195" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG { 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-cts-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "instance": "10.142.0.107:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "instance": "10.142.0.77:9100" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "11480836.733333332" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974984" 03:23:04 INFO [http_cmd]: http status code OK 03:23:04 DEBUG --- status code --- 03:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:04 DEBUG --- http response --- 03:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:04 DEBUG ----- output ----- 03:23:04 DEBUG { 03:23:04 DEBUG "status": "success", 03:23:04 DEBUG "data": { 03:23:04 DEBUG "resultType": "vector", 03:23:04 DEBUG "result": [ 03:23:04 DEBUG { 03:23:04 DEBUG "metric": { 03:23:04 DEBUG "pod": "ds-idrepo-1" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG [...] 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG 03:23:04 DEBUG "pod": "ds-idrepo-0" 03:23:04 DEBUG }, 03:23:04 DEBUG "value": [ 03:23:04 DEBUG 1712974984, 03:23:04 DEBUG "0" 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG ] 03:23:04 DEBUG } 03:23:04 DEBUG } 03:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974985" 03:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974985" 03:23:05 INFO [http_cmd]: http status code OK 03:23:05 INFO [http_cmd]: http status code OK 03:23:05 DEBUG --- status code --- 03:23:05 DEBUG --- status code --- 03:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:05 DEBUG --- http response --- 03:23:05 DEBUG --- http response --- 03:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:23:05 DEBUG ----- output ----- 03:23:05 DEBUG ----- output ----- 03:23:05 DEBUG { 03:23:05 DEBUG { 03:23:05 DEBUG "status": "success", 03:23:05 DEBUG "status": "success", 03:23:05 DEBUG "data": { 03:23:05 DEBUG "data": { 03:23:05 DEBUG "resultType": "vector", 03:23:05 DEBUG "resultType": "vector", 03:23:05 DEBUG "result": [ 03:23:05 DEBUG "result": [ 03:23:05 DEBUG { 03:23:05 DEBUG { 03:23:05 DEBUG "metric": { 03:23:05 DEBUG "metric": { 03:23:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:23:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:23:05 DEBUG }, 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG [...] 03:23:05 DEBUG [...] 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:05 DEBUG }, 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 1712974985, 03:23:05 DEBUG 1712974985, 03:23:05 DEBUG "1.978502192272297" 03:23:05 DEBUG "0" 03:23:05 DEBUG ] 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:05 DEBUG ] 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712974985" 03:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712974985" 03:23:05 INFO [http_cmd]: http status code OK 03:23:05 DEBUG --- status code --- 03:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:05 DEBUG --- http response --- 03:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:05 DEBUG ----- output ----- 03:23:05 DEBUG { 03:23:05 DEBUG "status": "success", 03:23:05 DEBUG "data": { 03:23:05 DEBUG "resultType": "vector", 03:23:05 DEBUG "result": [ 03:23:05 DEBUG { 03:23:05 DEBUG "metric": { 03:23:05 DEBUG "instance": "10.142.0.107:9100" 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG [...] 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG "instance": "10.142.0.77:9100" 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 1712974985, 03:23:05 DEBUG "0.01933333333333399" 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:05 INFO [http_cmd]: http status code OK 03:23:05 DEBUG --- status code --- 03:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:05 DEBUG --- http response --- 03:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:23:05 DEBUG ----- output ----- 03:23:05 DEBUG { 03:23:05 DEBUG "status": "success", 03:23:05 DEBUG "data": { 03:23:05 DEBUG "resultType": "vector", 03:23:05 DEBUG "result": [ 03:23:05 DEBUG { 03:23:05 DEBUG "metric": { 03:23:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG [...] 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 1712974985, 03:23:05 DEBUG "6240878592" 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712974985" 03:23:05 INFO [http_cmd]: http status code OK 03:23:05 DEBUG --- status code --- 03:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:23:05 DEBUG --- http response --- 03:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:23:05 DEBUG ----- output ----- 03:23:05 DEBUG { 03:23:05 DEBUG "status": "success", 03:23:05 DEBUG "data": { 03:23:05 DEBUG "resultType": "vector", 03:23:05 DEBUG "result": [ 03:23:05 DEBUG { 03:23:05 DEBUG "metric": { 03:23:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG [...] 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG 03:23:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:23:05 DEBUG }, 03:23:05 DEBUG "value": [ 03:23:05 DEBUG 1712974985, 03:23:05 DEBUG "0" 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG ] 03:23:05 DEBUG } 03:23:05 DEBUG } 03:23:19 INFO 03:23:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:23:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:23:19 INFO [loop_until]: OK (rc = 0) 03:23:19 DEBUG --- stdout --- 03:23:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 1948m 8492Mi am-6d96779b5d-qbg7l 1668m 8428Mi am-6d96779b5d-zn69c 1899m 8424Mi ds-cts-0 5072m 13126Mi ds-cts-1 5142m 13514Mi ds-cts-2 4716m 13501Mi ds-idrepo-0 942m 5594Mi ds-idrepo-1 376m 5523Mi ds-idrepo-2 374m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 1m 68Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 1m 3466Mi 03:23:19 DEBUG --- stderr --- 03:23:19 DEBUG 03:23:25 INFO 03:23:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:23:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:23:25 INFO [loop_until]: OK (rc = 0) 03:23:25 DEBUG --- stdout --- 03:23:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4727Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 1862Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1757m 5% 10359Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 110m 0% 3085Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2007m 6% 10249Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4339Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2107m 7% 10362Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 473m 2% 6310Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4938m 31% 14459Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 5286m 33% 14433Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 479m 3% 6710Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 5278m 33% 14073Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 960m 6% 6393Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 7217Mi 12% 03:23:25 DEBUG --- stderr --- 03:23:25 DEBUG 03:23:43 WARNING File /tmp/lodemon_data-20240412_220143/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [13/Apr/2024 03:23:43] "GET /monitoring/average?start_time=20240413_012259&stop_time=20240413_022142 HTTP/1.1" 200 - 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "instance": "10.142.0.107:9100" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "instance": "10.142.0.77:9100" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "722451432448" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "ds-idrepo-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "ds-idrepo-0" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG { 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG { 03:24:04 DEBUG "data": { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:04 DEBUG }, 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:04 DEBUG }, 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0" 03:24:04 DEBUG "1750.5666666666666" 03:24:04 DEBUG ] 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975044" 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "ds-idrepo-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "ds-cts-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0.0004" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 ERROR [http_cmd]: ERROR 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 ERROR --- status code --- 03:24:04 DEBUG --- status code --- 03:24:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 ERROR --- http response --- 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:24:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "ds-idrepo-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "ds-cts-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "29.166666666666668" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0.023" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "136371.5" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "instance": "10.142.0.107:9100" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "instance": "10.142.0.77:9100" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0.11820000000000021" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG { 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "ds-idrepo-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "ds-idrepo-0" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "2218661.3934806115" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0.025971263024547043" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975044" 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "instance": "10.142.0.107:9100" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "instance": "10.142.0.77:9100" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "11535934.933333334" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "ds-idrepo-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "ds-cts-1" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "0" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:04 INFO [http_cmd]: http status code OK 03:24:04 DEBUG --- status code --- 03:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:04 DEBUG --- http response --- 03:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:24:04 DEBUG ----- output ----- 03:24:04 DEBUG { 03:24:04 DEBUG "status": "success", 03:24:04 DEBUG "data": { 03:24:04 DEBUG "resultType": "vector", 03:24:04 DEBUG "result": [ 03:24:04 DEBUG { 03:24:04 DEBUG "metric": { 03:24:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG [...] 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG 03:24:04 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:24:04 DEBUG }, 03:24:04 DEBUG "value": [ 03:24:04 DEBUG 1712975044, 03:24:04 DEBUG "2910.798702301869" 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG ] 03:24:04 DEBUG } 03:24:04 DEBUG } 03:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975044" 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "pod": "ds-idrepo-1" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "pod": "ds-cts-1" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975044, 03:24:05 DEBUG "0" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975045" 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975045" 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "0" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975045" 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "instance": "10.142.0.107:9100" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "instance": "10.142.0.77:9100" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "11529623.433333334" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "pod": "ds-idrepo-1" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "pod": "ds-idrepo-0" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "0" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975045" 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975045" 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975045" 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "0" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "instance": "10.142.0.107:9100" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "instance": "10.142.0.77:9100" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "0.03266666666666751" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "0.0047996910242542965" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975045" 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "6241288192" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975045" 03:24:05 INFO [http_cmd]: http status code OK 03:24:05 DEBUG --- status code --- 03:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:24:05 DEBUG --- http response --- 03:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:24:05 DEBUG ----- output ----- 03:24:05 DEBUG { 03:24:05 DEBUG "status": "success", 03:24:05 DEBUG "data": { 03:24:05 DEBUG "resultType": "vector", 03:24:05 DEBUG "result": [ 03:24:05 DEBUG { 03:24:05 DEBUG "metric": { 03:24:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG [...] 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG 03:24:05 DEBUG "pod": "lodemon-8dc49bd56-qr24k" 03:24:05 DEBUG }, 03:24:05 DEBUG "value": [ 03:24:05 DEBUG 1712975045, 03:24:05 DEBUG "0" 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG ] 03:24:05 DEBUG } 03:24:05 DEBUG } 03:24:19 INFO 03:24:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:24:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:24:19 INFO [loop_until]: OK (rc = 0) 03:24:19 DEBUG --- stdout --- 03:24:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 1962m 8491Mi am-6d96779b5d-qbg7l 1674m 8431Mi am-6d96779b5d-zn69c 1941m 8423Mi ds-cts-0 4660m 13304Mi ds-cts-1 4798m 13392Mi ds-cts-2 4741m 13354Mi ds-idrepo-0 897m 5594Mi ds-idrepo-1 381m 5523Mi ds-idrepo-2 385m 5862Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1120Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 8m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 3m 3466Mi 03:24:19 DEBUG --- stderr --- 03:24:19 DEBUG 03:24:25 INFO 03:24:25 INFO [loop_until]: kubectl --namespace=xlou top node 03:24:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:24:26 INFO [loop_until]: OK (rc = 0) 03:24:26 DEBUG --- stdout --- 03:24:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4732Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 1861Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1731m 5% 10356Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 127m 0% 3117Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2085m 6% 10258Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 4354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2132m 7% 10364Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 443m 2% 6312Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4824m 30% 14317Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 4695m 29% 14313Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6711Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 4683m 29% 14271Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 1018m 6% 6392Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 7218Mi 12% 03:24:26 DEBUG --- stderr --- 03:24:26 DEBUG 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "instance": "10.142.0.107:9100" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "instance": "10.142.0.77:9100" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "723307746304" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0.03333333333333333" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "ds-idrepo-1" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "ds-idrepo-0" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "ds-idrepo-1" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "ds-cts-1" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 ERROR [http_cmd]: ERROR 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 ERROR --- status code --- 03:25:04 DEBUG --- status code --- 03:25:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 ERROR --- http response --- 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:25:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "1739.3333333333333" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0.03333333333333333" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "ds-idrepo-1" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "ds-cts-1" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "28.433333333333334" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0.023" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "31406" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "ds-idrepo-1" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "ds-idrepo-0" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "instance": "10.142.0.107:9100" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "instance": "10.142.0.77:9100" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "0.017900000000000204" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975104" 03:25:04 INFO [http_cmd]: http status code OK 03:25:04 DEBUG --- status code --- 03:25:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:04 DEBUG --- http response --- 03:25:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:25:04 DEBUG ----- output ----- 03:25:04 DEBUG { 03:25:04 DEBUG "status": "success", 03:25:04 DEBUG "data": { 03:25:04 DEBUG "resultType": "vector", 03:25:04 DEBUG "result": [ 03:25:04 DEBUG { 03:25:04 DEBUG "metric": { 03:25:04 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG [...] 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG 03:25:04 DEBUG "pod": "ds-cts-2" 03:25:04 DEBUG }, 03:25:04 DEBUG "value": [ 03:25:04 DEBUG 1712975104, 03:25:04 DEBUG "479545.5834246823" 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG ] 03:25:04 DEBUG } 03:25:04 DEBUG } 03:25:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975104" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "ds-cts-2" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975104, 03:25:05 DEBUG "1745444.4009395894" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975105" 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975104, 03:25:05 DEBUG "0.029763042105203266" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "instance": "10.142.0.107:9100" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "instance": "10.142.0.77:9100" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "11517796.366666667" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "ds-idrepo-1" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "ds-cts-1" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "ds-idrepo-1" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "ds-cts-1" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "ds-idrepo-1" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "ds-idrepo-0" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975105" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0.16666666666666666" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "instance": "10.142.0.107:9100" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "instance": "10.142.0.77:9100" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "11511312.566666666" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975105" 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0.03333333333333333" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975105" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "ds-cts-2" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "2.795808632995985" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "instance": "10.142.0.107:9100" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "instance": "10.142.0.77:9100" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0.007000000000000266" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "6241759232" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975105" 03:25:05 INFO [http_cmd]: http status code OK 03:25:05 DEBUG --- status code --- 03:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:25:05 DEBUG --- http response --- 03:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:25:05 DEBUG ----- output ----- 03:25:05 DEBUG { 03:25:05 DEBUG "status": "success", 03:25:05 DEBUG "data": { 03:25:05 DEBUG "resultType": "vector", 03:25:05 DEBUG "result": [ 03:25:05 DEBUG { 03:25:05 DEBUG "metric": { 03:25:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG [...] 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG 03:25:05 DEBUG "pod": "ds-cts-2" 03:25:05 DEBUG }, 03:25:05 DEBUG "value": [ 03:25:05 DEBUG 1712975105, 03:25:05 DEBUG "0" 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG ] 03:25:05 DEBUG } 03:25:05 DEBUG } 03:25:19 INFO 03:25:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:25:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:25:19 INFO [loop_until]: OK (rc = 0) 03:25:19 DEBUG --- stdout --- 03:25:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 196m 8490Mi am-6d96779b5d-qbg7l 64m 8428Mi am-6d96779b5d-zn69c 47m 8425Mi ds-cts-0 18m 13329Mi ds-cts-1 24m 13494Mi ds-cts-2 1186m 13381Mi ds-idrepo-0 508m 5595Mi ds-idrepo-1 284m 5522Mi ds-idrepo-2 307m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1121Mi idm-b85f9d978-dz57p 3m 1122Mi lodemon-8dc49bd56-qr24k 5m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 480m 4198Mi 03:25:19 DEBUG --- stderr --- 03:25:19 DEBUG 03:25:26 INFO 03:25:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:25:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:25:26 INFO [loop_until]: OK (rc = 0) 03:25:26 DEBUG --- stdout --- 03:25:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4732Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 1874Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 112m 0% 10360Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 3120Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 142m 0% 10250Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 148m 0% 4348Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 69m 0% 10353Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 311m 1% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 533m 3% 14336Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 81m 0% 14425Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 286m 1% 6726Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 525m 3% 14297Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 469m 2% 6397Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 857m 5% 7971Mi 13% 03:25:26 DEBUG --- stderr --- 03:25:26 DEBUG 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975164" 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "pod": "ds-idrepo-1" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "ds-idrepo-0" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "0" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "instance": "10.142.0.107:9100" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "instance": "10.142.0.77:9100" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "723385926656" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975164" 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "0" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "data": { 03:26:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG }, 03:26:04 DEBUG "result": [ 03:26:04 DEBUG "value": [ 03:26:04 DEBUG { 03:26:04 DEBUG 03:26:04 DEBUG "metric": { 03:26:04 DEBUG 03:26:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:04 DEBUG 03:26:04 DEBUG }, 03:26:04 DEBUG [...] 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:04 DEBUG [...] 03:26:04 DEBUG }, 03:26:04 DEBUG 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG 03:26:04 DEBUG "0" 03:26:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:04 DEBUG ] 03:26:04 DEBUG }, 03:26:04 DEBUG } 03:26:04 DEBUG "value": [ 03:26:04 DEBUG ] 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG } 03:26:04 DEBUG "0" 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 ERROR [http_cmd]: ERROR 03:26:04 DEBUG --- status code --- 03:26:04 ERROR --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 ERROR --- http response --- 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:26:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:26:04 DEBUG ----- output ----- 03:26:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "pod": "ds-idrepo-1" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "ds-cts-1" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "0" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "0.023" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "pod": "ds-idrepo-1" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "ds-cts-1" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "29.5" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:04 INFO [http_cmd]: http status code OK 03:26:04 DEBUG --- status code --- 03:26:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:04 DEBUG --- http response --- 03:26:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:04 DEBUG ----- output ----- 03:26:04 DEBUG { 03:26:04 DEBUG "status": "success", 03:26:04 DEBUG "data": { 03:26:04 DEBUG "resultType": "vector", 03:26:04 DEBUG "result": [ 03:26:04 DEBUG { 03:26:04 DEBUG "metric": { 03:26:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG [...] 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG 03:26:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:04 DEBUG }, 03:26:04 DEBUG "value": [ 03:26:04 DEBUG 1712975164, 03:26:04 DEBUG "2" 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG ] 03:26:04 DEBUG } 03:26:04 DEBUG } 03:26:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975164" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "ds-idrepo-1" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "ds-idrepo-0" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975164, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "instance": "10.142.0.107:9100" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "instance": "10.142.0.77:9100" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0.0006666666666636918" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "420309.9402465393" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "1.1415573103679313" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "ds-idrepo-1" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "ds-cts-1" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "instance": "10.142.0.107:9100" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "instance": "10.142.0.77:9100" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "31009.9" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "ds-idrepo-1" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "ds-cts-1" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "ds-idrepo-1" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "ds-idrepo-0" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:05 DEBUG 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG [...] 03:26:05 DEBUG { 03:26:05 DEBUG 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG 03:26:05 DEBUG "data": { 03:26:05 DEBUG 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:05 DEBUG "result": [ 03:26:05 DEBUG }, 03:26:05 DEBUG { 03:26:05 DEBUG "value": [ 03:26:05 DEBUG "metric": { 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "instance": "10.142.0.107:9100" 03:26:05 DEBUG "0" 03:26:05 DEBUG }, 03:26:05 DEBUG ] 03:26:05 DEBUG "value": [ 03:26:05 DEBUG } 03:26:05 DEBUG 03:26:05 DEBUG ] 03:26:05 DEBUG 03:26:05 DEBUG } 03:26:05 DEBUG 03:26:05 DEBUG } 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "instance": "10.142.0.77:9100" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "24072.866666666665" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975165" 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "6243811328" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "instance": "10.142.0.107:9100" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG "instance": "10.142.0.77:9100" 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG }, 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG "value": [ 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0.0003333333333330302" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "2.0081282850195707" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975165" 03:26:05 INFO [http_cmd]: http status code OK 03:26:05 DEBUG --- status code --- 03:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:26:05 DEBUG --- http response --- 03:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:26:05 DEBUG ----- output ----- 03:26:05 DEBUG { 03:26:05 DEBUG "status": "success", 03:26:05 DEBUG "data": { 03:26:05 DEBUG "resultType": "vector", 03:26:05 DEBUG "result": [ 03:26:05 DEBUG { 03:26:05 DEBUG "metric": { 03:26:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG [...] 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG 03:26:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:26:05 DEBUG }, 03:26:05 DEBUG "value": [ 03:26:05 DEBUG 1712975165, 03:26:05 DEBUG "0" 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG ] 03:26:05 DEBUG } 03:26:05 DEBUG } 03:26:19 INFO 03:26:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:26:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:26:19 INFO [loop_until]: OK (rc = 0) 03:26:19 DEBUG --- stdout --- 03:26:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8416Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 16m 13329Mi ds-cts-1 19m 13494Mi ds-cts-2 17m 13381Mi ds-idrepo-0 14m 5596Mi ds-idrepo-1 12m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1122Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 988m 4216Mi 03:26:19 DEBUG --- stderr --- 03:26:19 DEBUG 03:26:26 INFO 03:26:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:26:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:26:26 INFO [loop_until]: OK (rc = 0) 03:26:26 DEBUG --- stdout --- 03:26:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4736Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 1867Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10348Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 121m 0% 3114Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 10246Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 10347Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6315Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 14333Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 81m 0% 14420Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6715Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 74m 0% 14295Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 70m 0% 6398Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 7979Mi 13% 03:26:26 DEBUG --- stderr --- 03:26:26 DEBUG 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG --- http response --- 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG { 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:04 DEBUG "data": { 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG { 03:27:04 DEBUG "result": [ 03:27:04 DEBUG { 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG { 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "data": { 03:27:04 DEBUG "metric": { 03:27:04 DEBUG "data": { 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG "result": [ 03:27:04 DEBUG }, 03:27:04 DEBUG "result": [ 03:27:04 DEBUG { 03:27:04 DEBUG "value": [ 03:27:04 DEBUG { 03:27:04 DEBUG "metric": { 03:27:04 DEBUG 03:27:04 DEBUG "metric": { 03:27:04 DEBUG "instance": "10.142.0.107:9100" 03:27:04 DEBUG 03:27:04 DEBUG "pod": "ds-idrepo-1" 03:27:04 DEBUG }, 03:27:04 DEBUG 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG [...] 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG [...] 03:27:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:04 DEBUG [...] 03:27:04 DEBUG 03:27:04 DEBUG }, 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG 03:27:04 DEBUG "instance": "10.142.0.77:9100" 03:27:04 DEBUG "0" 03:27:04 DEBUG "pod": "ds-idrepo-0" 03:27:04 DEBUG }, 03:27:04 DEBUG ] 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG } 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG ] 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG "723387458560" 03:27:04 DEBUG } 03:27:04 DEBUG "0" 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 DEBUG ] 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975224" 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG { 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "data": { 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG "result": [ 03:27:04 DEBUG { 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:04 DEBUG "metric": { 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:04 DEBUG { 03:27:04 DEBUG }, 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "value": [ 03:27:04 DEBUG "data": { 03:27:04 DEBUG 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG 03:27:04 DEBUG "result": [ 03:27:04 DEBUG 03:27:04 DEBUG { 03:27:04 DEBUG [...] 03:27:04 DEBUG "metric": { 03:27:04 DEBUG 03:27:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:04 DEBUG 03:27:04 DEBUG }, 03:27:04 DEBUG 03:27:04 DEBUG "value": [ 03:27:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:04 DEBUG 03:27:04 DEBUG }, 03:27:04 DEBUG 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG [...] 03:27:04 DEBUG "0" 03:27:04 DEBUG 03:27:04 DEBUG ] 03:27:04 DEBUG 03:27:04 DEBUG } 03:27:04 DEBUG 03:27:04 DEBUG ] 03:27:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:04 DEBUG } 03:27:04 DEBUG }, 03:27:04 DEBUG } 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG "0" 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG { 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "data": { 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG "result": [ 03:27:04 DEBUG { 03:27:04 DEBUG "metric": { 03:27:04 DEBUG "pod": "ds-idrepo-1" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG [...] 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG "pod": "ds-cts-1" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG "0" 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 ERROR [http_cmd]: ERROR 03:27:04 ERROR --- status code --- 03:27:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:27:04 ERROR --- http response --- 03:27:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:27:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975224" 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG { 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "data": { 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG "result": [ 03:27:04 DEBUG { 03:27:04 DEBUG "metric": { 03:27:04 DEBUG "pod": "ds-idrepo-1" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG [...] 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG "pod": "ds-cts-1" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG "28.766666666666666" 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG { 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "data": { 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG "result": [ 03:27:04 DEBUG { 03:27:04 DEBUG "metric": { 03:27:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 INFO [http_cmd]: http status code OK 03:27:04 DEBUG 03:27:04 DEBUG --- status code --- 03:27:04 DEBUG 03:27:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:04 DEBUG [...] 03:27:04 DEBUG --- http response --- 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:04 DEBUG }, 03:27:04 DEBUG ----- output ----- 03:27:04 DEBUG "value": [ 03:27:04 DEBUG { 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG "status": "success", 03:27:04 DEBUG "2" 03:27:04 DEBUG "data": { 03:27:04 DEBUG ] 03:27:04 DEBUG "resultType": "vector", 03:27:04 DEBUG } 03:27:04 DEBUG "result": [ 03:27:04 DEBUG ] 03:27:04 DEBUG { 03:27:04 DEBUG } 03:27:04 DEBUG "metric": { 03:27:04 DEBUG } 03:27:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG [...] 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG 03:27:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:04 DEBUG }, 03:27:04 DEBUG "value": [ 03:27:04 DEBUG 1712975224, 03:27:04 DEBUG "0.023" 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG ] 03:27:04 DEBUG } 03:27:04 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "ds-idrepo-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "ds-idrepo-0" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "instance": "10.142.0.107:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "instance": "10.142.0.77:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0.0009666666666689139" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "ds-idrepo-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "ds-cts-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "ds-idrepo-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "idm-b85f9d978-dz57p" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "598.261885634996" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975225" 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975225" 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "1.0299973700075893" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "ds-idrepo-1" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG }, 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG "value": [ 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "ds-cts-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "instance": "10.142.0.107:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "instance": "10.142.0.77:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "30599.366666666665" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "ds-idrepo-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "ds-idrepo-0" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "instance": "10.142.0.107:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "instance": "10.142.0.77:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "24332.13333333333" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975225" 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "6243782656" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "ds-idrepo-1" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975225" 03:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975225" 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "instance": "10.142.0.107:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "instance": "10.142.0.77:9100" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:05 INFO [http_cmd]: http status code OK 03:27:05 DEBUG --- status code --- 03:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:27:05 DEBUG --- http response --- 03:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:27:05 DEBUG ----- output ----- 03:27:05 DEBUG { 03:27:05 DEBUG "status": "success", 03:27:05 DEBUG "data": { 03:27:05 DEBUG "resultType": "vector", 03:27:05 DEBUG "result": [ 03:27:05 DEBUG { 03:27:05 DEBUG "metric": { 03:27:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG [...] 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG 03:27:05 DEBUG "pod": "idm-b85f9d978-dz57p" 03:27:05 DEBUG }, 03:27:05 DEBUG "value": [ 03:27:05 DEBUG 1712975225, 03:27:05 DEBUG "0" 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG ] 03:27:05 DEBUG } 03:27:05 DEBUG } 03:27:19 INFO 03:27:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:27:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:27:19 INFO [loop_until]: OK (rc = 0) 03:27:19 DEBUG --- stdout --- 03:27:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 17m 13329Mi ds-cts-1 19m 13494Mi ds-cts-2 18m 13381Mi ds-idrepo-0 16m 5595Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1122Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 1008m 4250Mi 03:27:19 DEBUG --- stderr --- 03:27:19 DEBUG 03:27:26 INFO 03:27:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:27:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:27:26 INFO [loop_until]: OK (rc = 0) 03:27:26 DEBUG --- stdout --- 03:27:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4740Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10348Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 3100Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 60m 0% 10235Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4365Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 10334Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6310Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 14332Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 77m 0% 14419Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6713Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 74m 0% 14298Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 72m 0% 6401Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 8017Mi 13% 03:27:26 DEBUG --- stderr --- 03:27:26 DEBUG 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 DEBUG --- status code --- 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 DEBUG --- http response --- 03:28:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975284" 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975284" 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 DEBUG --- status code --- 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 DEBUG --- http response --- 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:04 DEBUG --- status code --- 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:04 DEBUG --- status code --- 03:28:04 DEBUG ----- output ----- 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 DEBUG { 03:28:04 DEBUG --- http response --- 03:28:04 DEBUG --- http response --- 03:28:04 DEBUG "status": "success", 03:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:04 DEBUG "data": { 03:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:04 DEBUG ----- output ----- 03:28:04 DEBUG "resultType": "vector", 03:28:04 DEBUG ----- output ----- 03:28:04 DEBUG { 03:28:04 DEBUG "result": [ 03:28:04 DEBUG { 03:28:04 DEBUG "status": "success", 03:28:04 DEBUG { 03:28:04 DEBUG "status": "success", 03:28:04 DEBUG "data": { 03:28:04 DEBUG "metric": { 03:28:04 DEBUG "data": { 03:28:04 DEBUG "resultType": "vector", 03:28:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:04 DEBUG "resultType": "vector", 03:28:04 DEBUG "result": [ 03:28:04 DEBUG }, 03:28:04 DEBUG "result": [ 03:28:04 DEBUG { 03:28:04 DEBUG "value": [ 03:28:04 DEBUG { 03:28:04 DEBUG "metric": { 03:28:04 DEBUG 03:28:04 DEBUG "metric": { 03:28:04 DEBUG "pod": "ds-idrepo-1" 03:28:04 DEBUG 03:28:04 DEBUG "instance": "10.142.0.107:9100" 03:28:04 DEBUG }, 03:28:04 DEBUG 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG [...] 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG [...] 03:28:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:04 DEBUG [...] 03:28:04 DEBUG 03:28:04 DEBUG }, 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 1712975284, 03:28:04 DEBUG 03:28:04 DEBUG "pod": "ds-idrepo-0" 03:28:04 DEBUG "0" 03:28:04 DEBUG "instance": "10.142.0.77:9100" 03:28:04 DEBUG }, 03:28:04 DEBUG ] 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG } 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 1712975284, 03:28:04 DEBUG ] 03:28:04 DEBUG 1712975284, 03:28:04 DEBUG "0" 03:28:04 DEBUG } 03:28:04 DEBUG "723389027328" 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 DEBUG ] 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 DEBUG --- status code --- 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 DEBUG --- http response --- 03:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:04 DEBUG ----- output ----- 03:28:04 DEBUG { 03:28:04 DEBUG "status": "success", 03:28:04 DEBUG "data": { 03:28:04 DEBUG "resultType": "vector", 03:28:04 DEBUG "result": [ 03:28:04 DEBUG { 03:28:04 DEBUG "metric": { 03:28:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG [...] 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 1712975284, 03:28:04 DEBUG "0.06666666666666667" 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 ERROR [http_cmd]: ERROR 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 ERROR --- status code --- 03:28:04 DEBUG --- status code --- 03:28:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 ERROR --- http response --- 03:28:04 DEBUG --- http response --- 03:28:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:28:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:04 DEBUG ----- output ----- 03:28:04 DEBUG { 03:28:04 DEBUG "status": "success", 03:28:04 DEBUG "data": { 03:28:04 DEBUG "resultType": "vector", 03:28:04 DEBUG "result": [ 03:28:04 DEBUG { 03:28:04 DEBUG "metric": { 03:28:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG [...] 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 1712975284, 03:28:04 DEBUG "0" 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 INFO [http_cmd]: http status code OK 03:28:04 DEBUG --- status code --- 03:28:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:04 DEBUG --- http response --- 03:28:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:28:04 DEBUG ----- output ----- 03:28:04 DEBUG { 03:28:04 DEBUG "status": "success", 03:28:04 DEBUG "data": { 03:28:04 DEBUG "resultType": "vector", 03:28:04 DEBUG "result": [ 03:28:04 DEBUG { 03:28:04 DEBUG "metric": { 03:28:04 DEBUG "pod": "ds-idrepo-1" 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG [...] 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG 03:28:04 DEBUG "pod": "ds-cts-1" 03:28:04 DEBUG }, 03:28:04 DEBUG "value": [ 03:28:04 DEBUG 1712975284, 03:28:04 DEBUG "0" 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG ] 03:28:04 DEBUG } 03:28:04 DEBUG } 03:28:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975284" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "ds-idrepo-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "ds-cts-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975284, 03:28:05 DEBUG "27.5" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG { 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0.023" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "1" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "instance": "10.142.0.107:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "instance": "10.142.0.77:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0.0012666666666679778" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "ds-idrepo-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "ds-idrepo-0" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "849.3027406964688" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "290575087.029905" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "ds-idrepo-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "ds-cts-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG { 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "1.030939827085411" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "ds-idrepo-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "ds-cts-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "instance": "10.142.0.107:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "instance": "10.142.0.77:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "31326.43333333333" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "ds-idrepo-1" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "ds-idrepo-0" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "instance": "10.142.0.107:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "instance": "10.142.0.77:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "27244.199999999997" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "1.0374505228405322" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "6244495360" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975285" 03:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975285" 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "instance": "10.142.0.107:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "instance": "10.142.0.77:9100" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "0.00033333333333492493" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:05 INFO [http_cmd]: http status code OK 03:28:05 DEBUG --- status code --- 03:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:28:05 DEBUG --- http response --- 03:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:28:05 DEBUG ----- output ----- 03:28:05 DEBUG { 03:28:05 DEBUG "status": "success", 03:28:05 DEBUG "data": { 03:28:05 DEBUG "resultType": "vector", 03:28:05 DEBUG "result": [ 03:28:05 DEBUG { 03:28:05 DEBUG "metric": { 03:28:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG [...] 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG 03:28:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:28:05 DEBUG }, 03:28:05 DEBUG "value": [ 03:28:05 DEBUG 1712975285, 03:28:05 DEBUG "283.1009135654896" 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG ] 03:28:05 DEBUG } 03:28:05 DEBUG } 03:28:19 INFO 03:28:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:28:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:28:19 INFO [loop_until]: OK (rc = 0) 03:28:19 DEBUG --- stdout --- 03:28:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 8m 8481Mi am-6d96779b5d-qbg7l 6m 8416Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 22m 12373Mi ds-cts-1 20m 13494Mi ds-cts-2 15m 13381Mi ds-idrepo-0 13m 5596Mi ds-idrepo-1 12m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 3m 1122Mi lodemon-8dc49bd56-qr24k 1m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 324m 4286Mi 03:28:19 DEBUG --- stderr --- 03:28:19 DEBUG 03:28:26 INFO 03:28:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:28:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:28:26 INFO [loop_until]: OK (rc = 0) 03:28:26 DEBUG --- stdout --- 03:28:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4747Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1863Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10335Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 92m 0% 3112Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 60m 0% 10227Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 10317Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6315Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 14333Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 73m 0% 14418Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6717Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 76m 0% 13345Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6397Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 578m 3% 8080Mi 13% 03:28:26 DEBUG --- stderr --- 03:28:26 DEBUG 03:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975344" 03:29:04 INFO [http_cmd]: http status code OK 03:29:04 DEBUG --- status code --- 03:29:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:04 DEBUG --- http response --- 03:29:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975344" 03:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975344" 03:29:04 INFO [http_cmd]: http status code OK 03:29:04 DEBUG --- status code --- 03:29:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:04 DEBUG --- http response --- 03:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:04 DEBUG ----- output ----- 03:29:04 DEBUG { 03:29:04 DEBUG "status": "success", 03:29:04 DEBUG "data": { 03:29:04 DEBUG "resultType": "vector", 03:29:04 DEBUG "result": [ 03:29:04 DEBUG { 03:29:04 DEBUG "metric": { 03:29:04 DEBUG "pod": "ds-idrepo-1" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG [...] 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG "pod": "ds-idrepo-0" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 1712975344, 03:29:04 DEBUG "0" 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG } 03:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975344" 03:29:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975344" 03:29:04 INFO [http_cmd]: http status code OK 03:29:04 DEBUG --- status code --- 03:29:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:04 DEBUG --- http response --- 03:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:04 DEBUG ----- output ----- 03:29:04 DEBUG { 03:29:04 DEBUG "status": "success", 03:29:04 DEBUG "data": { 03:29:04 DEBUG "resultType": "vector", 03:29:04 DEBUG "result": [ 03:29:04 DEBUG { 03:29:04 DEBUG "metric": { 03:29:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG [...] 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 1712975344, 03:29:04 DEBUG "0" 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG } 03:29:04 INFO [http_cmd]: http status code OK 03:29:04 DEBUG --- status code --- 03:29:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:04 DEBUG --- http response --- 03:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:04 DEBUG ----- output ----- 03:29:04 DEBUG { 03:29:04 DEBUG "status": "success", 03:29:04 DEBUG "data": { 03:29:04 DEBUG "resultType": "vector", 03:29:04 DEBUG "result": [ 03:29:04 DEBUG { 03:29:04 DEBUG "metric": { 03:29:04 DEBUG "instance": "10.142.0.107:9100" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG [...] 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG "instance": "10.142.0.77:9100" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 1712975344, 03:29:04 DEBUG "723390696448" 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG } 03:29:04 INFO [http_cmd]: http status code OK 03:29:04 DEBUG --- status code --- 03:29:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:04 DEBUG --- http response --- 03:29:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:04 DEBUG ----- output ----- 03:29:04 DEBUG { 03:29:04 DEBUG "status": "success", 03:29:04 DEBUG "data": { 03:29:04 DEBUG "resultType": "vector", 03:29:04 DEBUG "result": [ 03:29:04 DEBUG { 03:29:04 DEBUG "metric": { 03:29:04 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG [...] 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG 03:29:04 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:04 DEBUG }, 03:29:04 DEBUG "value": [ 03:29:04 DEBUG 1712975344, 03:29:04 DEBUG "0" 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG ] 03:29:04 DEBUG } 03:29:04 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 ERROR [http_cmd]: ERROR 03:29:05 ERROR --- status code --- 03:29:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:29:05 ERROR --- http response --- 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG --- status code --- 03:29:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG [...] 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG 03:29:05 DEBUG { 03:29:05 DEBUG 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG 03:29:05 DEBUG "data": { 03:29:05 DEBUG "pod": "ds-cts-1" 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG }, 03:29:05 DEBUG "result": [ 03:29:05 DEBUG "value": [ 03:29:05 DEBUG { 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "0" 03:29:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:05 DEBUG ] 03:29:05 DEBUG }, 03:29:05 DEBUG } 03:29:05 DEBUG "value": [ 03:29:05 DEBUG ] 03:29:05 DEBUG 03:29:05 DEBUG } 03:29:05 DEBUG 03:29:05 DEBUG } 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "ds-cts-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "28.866666666666667" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0.0235" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "instance": "10.142.0.107:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "instance": "10.142.0.77:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0.0010333333333396885" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "ds-idrepo-0" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:29:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:29:05 DEBUG }, 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975345" 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "ds-cts-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0.8065155065110238" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "ds-idrepo-0" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "ds-cts-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "instance": "10.142.0.107:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "instance": "10.142.0.77:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "30578.5" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "instance": "10.142.0.107:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "instance": "10.142.0.77:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "24214.8" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975345" 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "6245756928" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "ds-idrepo-1" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "1243416.7064292312" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975345" 03:29:05 INFO [http_cmd]: http status code OK 03:29:05 DEBUG --- status code --- 03:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:29:05 DEBUG --- http response --- 03:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:29:05 DEBUG ----- output ----- 03:29:05 DEBUG { 03:29:05 DEBUG "status": "success", 03:29:05 DEBUG "data": { 03:29:05 DEBUG "resultType": "vector", 03:29:05 DEBUG "result": [ 03:29:05 DEBUG { 03:29:05 DEBUG "metric": { 03:29:05 DEBUG "instance": "10.142.0.107:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG [...] 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG 03:29:05 DEBUG "instance": "10.142.0.77:9100" 03:29:05 DEBUG }, 03:29:05 DEBUG "value": [ 03:29:05 DEBUG 1712975345, 03:29:05 DEBUG "0" 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG ] 03:29:05 DEBUG } 03:29:05 DEBUG } 03:29:19 INFO 03:29:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:29:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:29:19 INFO [loop_until]: OK (rc = 0) 03:29:19 DEBUG --- stdout --- 03:29:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 7m 8413Mi ds-cts-0 19m 12373Mi ds-cts-1 20m 13494Mi ds-cts-2 18m 13381Mi ds-idrepo-0 14m 5595Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1121Mi idm-b85f9d978-dz57p 3m 1122Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 1019m 4286Mi 03:29:19 DEBUG --- stderr --- 03:29:19 DEBUG 03:29:26 INFO 03:29:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:29:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:29:26 INFO [loop_until]: OK (rc = 0) 03:29:26 DEBUG --- stdout --- 03:29:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4746Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1864Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 10323Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 113m 0% 3111Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 10233Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 4353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 10329Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6325Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 68m 0% 14336Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 81m 0% 14421Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6719Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 76m 0% 13340Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6400Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1035m 6% 8100Mi 13% 03:29:26 DEBUG --- stderr --- 03:29:26 DEBUG 03:30:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975404" 03:30:04 INFO [http_cmd]: http status code OK 03:30:04 DEBUG --- status code --- 03:30:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:04 DEBUG --- http response --- 03:30:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-idrepo-0" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975405" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG }, 03:30:05 DEBUG "instance": "10.142.0.107:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-cts-1" 03:30:05 DEBUG 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG }, 03:30:05 DEBUG "instance": "10.142.0.77:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG "723392277504" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 ERROR [http_cmd]: ERROR 03:30:05 ERROR --- status code --- 03:30:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:30:05 ERROR --- http response --- 03:30:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:30:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-cts-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "29.266666666666666" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG }, 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG }, 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0.024" 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "instance": "10.142.0.107:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "instance": "10.142.0.77:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0.0010333333333377936" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "1.0213730563228425" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-idrepo-0" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "data": { 03:30:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG }, 03:30:05 DEBUG "result": [ 03:30:05 DEBUG "value": [ 03:30:05 DEBUG { 03:30:05 DEBUG 03:30:05 DEBUG "metric": { 03:30:05 DEBUG 03:30:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:30:05 DEBUG 03:30:05 DEBUG }, 03:30:05 DEBUG [...] 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:30:05 DEBUG [...] 03:30:05 DEBUG }, 03:30:05 DEBUG 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG 03:30:05 DEBUG "0" 03:30:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:30:05 DEBUG ] 03:30:05 DEBUG }, 03:30:05 DEBUG } 03:30:05 DEBUG "value": [ 03:30:05 DEBUG ] 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG } 03:30:05 DEBUG "52455.61750230733" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG } 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG ] 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG } 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:30:05 DEBUG } 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-cts-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-idrepo-0" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "ds-idrepo-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "ds-cts-1" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "instance": "10.142.0.107:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "instance": "10.142.0.77:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "30588.166666666664" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975405" 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "instance": "10.142.0.107:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "instance": "10.142.0.77:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "24108.566666666666" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "1.3965046309979319" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "6245715968" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "instance": "10.142.0.107:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "instance": "10.142.0.77:9100" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "0.000333333333333267" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975405" 03:30:05 INFO [http_cmd]: http status code OK 03:30:05 DEBUG --- status code --- 03:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:30:05 DEBUG --- http response --- 03:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:30:05 DEBUG ----- output ----- 03:30:05 DEBUG { 03:30:05 DEBUG "status": "success", 03:30:05 DEBUG "data": { 03:30:05 DEBUG "resultType": "vector", 03:30:05 DEBUG "result": [ 03:30:05 DEBUG { 03:30:05 DEBUG "metric": { 03:30:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG [...] 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG 03:30:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:30:05 DEBUG }, 03:30:05 DEBUG "value": [ 03:30:05 DEBUG 1712975405, 03:30:05 DEBUG "2232128.141904349" 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG ] 03:30:05 DEBUG } 03:30:05 DEBUG } 03:30:19 INFO 03:30:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:30:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:30:19 INFO [loop_until]: OK (rc = 0) 03:30:19 DEBUG --- stdout --- 03:30:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 17m 12373Mi ds-cts-1 21m 13494Mi ds-cts-2 18m 13381Mi ds-idrepo-0 14m 5595Mi ds-idrepo-1 12m 5523Mi ds-idrepo-2 13m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1121Mi idm-b85f9d978-dz57p 4m 1122Mi lodemon-8dc49bd56-qr24k 5m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 980m 4389Mi 03:30:19 DEBUG --- stderr --- 03:30:19 DEBUG 03:30:26 INFO 03:30:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:30:26 INFO [loop_until]: OK (rc = 0) 03:30:26 DEBUG --- stdout --- 03:30:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4744Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1861Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10324Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 117m 0% 3117Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 10226Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 4380Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 10328Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 68m 0% 6320Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 14336Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 81m 0% 14423Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 71m 0% 6720Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 74m 0% 13344Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6399Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1047m 6% 8183Mi 13% 03:30:26 DEBUG --- stderr --- 03:30:26 DEBUG 03:31:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975464" 03:31:04 INFO [http_cmd]: http status code OK 03:31:04 DEBUG --- status code --- 03:31:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:04 DEBUG --- http response --- 03:31:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-idrepo-0" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG { 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG "0" 03:31:05 DEBUG { 03:31:05 DEBUG ] 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG } 03:31:05 DEBUG "data": { 03:31:05 DEBUG ] 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG } 03:31:05 DEBUG "result": [ 03:31:05 DEBUG } 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG "instance": "10.142.0.107:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG "instance": "10.142.0.77:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG "723393862656" 03:31:05 DEBUG ] 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 ERROR [http_cmd]: ERROR 03:31:05 ERROR --- status code --- 03:31:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:31:05 ERROR --- http response --- 03:31:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:31:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "28.966666666666665" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0.024" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "instance": "10.142.0.107:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "instance": "10.142.0.77:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0.0006999999999905526" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "485431434.9034272" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0.9936831345883632" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-idrepo-0" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "data": { 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG }, 03:31:05 DEBUG "result": [ 03:31:05 DEBUG "value": [ 03:31:05 DEBUG { 03:31:05 DEBUG 03:31:05 DEBUG "metric": { 03:31:05 DEBUG 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG }, 03:31:05 DEBUG [...] 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG 03:31:05 DEBUG }, 03:31:05 DEBUG { 03:31:05 DEBUG [...] 03:31:05 DEBUG "value": [ 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "data": { 03:31:05 DEBUG 03:31:05 DEBUG "0" 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG 03:31:05 DEBUG ] 03:31:05 DEBUG "result": [ 03:31:05 DEBUG "pod": "ds-idrepo-0" 03:31:05 DEBUG } 03:31:05 DEBUG { 03:31:05 DEBUG }, 03:31:05 DEBUG ] 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "value": [ 03:31:05 DEBUG } 03:31:05 DEBUG "pod": "ds-idrepo-1" 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG } 03:31:05 DEBUG }, 03:31:05 DEBUG "0" 03:31:05 DEBUG "value": [ 03:31:05 DEBUG ] 03:31:05 DEBUG 03:31:05 DEBUG } 03:31:05 DEBUG 03:31:05 DEBUG ] 03:31:05 DEBUG 03:31:05 DEBUG } 03:31:05 DEBUG [...] 03:31:05 DEBUG } 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "15316.614635150432" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "instance": "10.142.0.107:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "instance": "10.142.0.77:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "30597.7" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "instance": "10.142.0.107:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "instance": "10.142.0.77:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "24163.63333333333" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0.016429790043947158" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "pod": "ds-cts-1" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "6245679104" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975465" 03:31:05 INFO [http_cmd]: http status code OK 03:31:05 DEBUG --- status code --- 03:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:31:05 DEBUG --- http response --- 03:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:31:05 DEBUG ----- output ----- 03:31:05 DEBUG { 03:31:05 DEBUG "status": "success", 03:31:05 DEBUG "data": { 03:31:05 DEBUG "resultType": "vector", 03:31:05 DEBUG "result": [ 03:31:05 DEBUG { 03:31:05 DEBUG "metric": { 03:31:05 DEBUG "instance": "10.142.0.107:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG [...] 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG 03:31:05 DEBUG "instance": "10.142.0.77:9100" 03:31:05 DEBUG }, 03:31:05 DEBUG "value": [ 03:31:05 DEBUG 1712975465, 03:31:05 DEBUG "0.0003333333333330302" 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG ] 03:31:05 DEBUG } 03:31:05 DEBUG } 03:31:19 INFO 03:31:19 INFO [loop_until]: kubectl --namespace=xlou top pods 03:31:19 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:31:19 INFO [loop_until]: OK (rc = 0) 03:31:19 DEBUG --- stdout --- 03:31:19 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 7m 8416Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 18m 12373Mi ds-cts-1 20m 13493Mi ds-cts-2 16m 13381Mi ds-idrepo-0 16m 5595Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 5m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 974m 4780Mi 03:31:19 DEBUG --- stderr --- 03:31:19 DEBUG 03:31:26 INFO 03:31:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:31:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:31:26 INFO [loop_until]: OK (rc = 0) 03:31:26 DEBUG --- stdout --- 03:31:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4745Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1852Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 10324Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 118m 0% 3119Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 56m 0% 10232Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 4366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 10314Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 14335Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 80m 0% 14422Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6716Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 76m 0% 13341Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 70m 0% 6398Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1064m 6% 8676Mi 14% 03:31:26 DEBUG --- stderr --- 03:31:26 DEBUG 03:32:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975524" 03:32:04 INFO [http_cmd]: http status code OK 03:32:04 DEBUG --- status code --- 03:32:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:04 DEBUG --- http response --- 03:32:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "ds-cts-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 ERROR [http_cmd]: ERROR 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 ERROR --- status code --- 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 ERROR --- http response --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG "instance": "10.142.0.107:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG "instance": "10.142.0.77:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG "723395525632" 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "ds-idrepo-0" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "ds-cts-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "29.266666666666666" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0.024" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "instance": "10.142.0.107:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "instance": "10.142.0.77:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0.0007666666666608534" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "236062264.234702" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "ds-cts-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "ds-idrepo-0" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "1.0215950945126029" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "data": { 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG "result": [ 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG "pod": "ds-idrepo-1" 03:32:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG [...] 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "ds-idrepo-0" 03:32:05 DEBUG "pod": "ds-cts-1" 03:32:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG "0" 03:32:05 DEBUG "4248.411772332426" 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "instance": "10.142.0.107:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "instance": "10.142.0.77:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "30978" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "instance": "10.142.0.107:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "instance": "10.142.0.77:9100" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "24696.066666666666" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "instance": "10.142.0.107:9100" 03:32:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "instance": "10.142.0.77:9100" 03:32:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:32:05 DEBUG }, 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0" 03:32:05 DEBUG "4528143.13496694" 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "0.3898517177618181" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975525" 03:32:05 INFO [http_cmd]: http status code OK 03:32:05 DEBUG --- status code --- 03:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:32:05 DEBUG --- http response --- 03:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:32:05 DEBUG ----- output ----- 03:32:05 DEBUG { 03:32:05 DEBUG "status": "success", 03:32:05 DEBUG "data": { 03:32:05 DEBUG "resultType": "vector", 03:32:05 DEBUG "result": [ 03:32:05 DEBUG { 03:32:05 DEBUG "metric": { 03:32:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG [...] 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG 03:32:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:32:05 DEBUG }, 03:32:05 DEBUG "value": [ 03:32:05 DEBUG 1712975525, 03:32:05 DEBUG "6244847616" 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG ] 03:32:05 DEBUG } 03:32:05 DEBUG } 03:32:20 INFO 03:32:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:32:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:32:20 INFO [loop_until]: OK (rc = 0) 03:32:20 DEBUG --- stdout --- 03:32:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 16m 8416Mi am-6d96779b5d-zn69c 7m 8413Mi ds-cts-0 18m 12373Mi ds-cts-1 17m 13493Mi ds-cts-2 20m 13381Mi ds-idrepo-0 13m 5596Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 4m 1121Mi idm-b85f9d978-dz57p 4m 1122Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 755m 4955Mi 03:32:20 DEBUG --- stderr --- 03:32:20 DEBUG 03:32:26 INFO 03:32:26 INFO [loop_until]: kubectl --namespace=xlou top node 03:32:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:32:27 INFO [loop_until]: OK (rc = 0) 03:32:27 DEBUG --- stdout --- 03:32:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 4746Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1866Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 67m 0% 10321Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 118m 0% 3110Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 61m 0% 10227Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 10314Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 72m 0% 14336Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 77m 0% 14425Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6719Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 13341Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6398Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1008m 6% 8842Mi 15% 03:32:27 DEBUG --- stderr --- 03:32:27 DEBUG 03:33:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975584" 03:33:04 INFO [http_cmd]: http status code OK 03:33:04 DEBUG --- status code --- 03:33:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:04 DEBUG --- http response --- 03:33:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "instance": "10.142.0.107:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "instance": "10.142.0.77:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "723397112832" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 ERROR [http_cmd]: ERROR 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 ERROR --- status code --- 03:33:05 DEBUG --- status code --- 03:33:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 ERROR --- http response --- 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:33:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "ds-cts-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "ds-idrepo-0" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "ds-cts-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "29.53333333333333" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0.024" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "instance": "10.142.0.107:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "instance": "10.142.0.77:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0.0010333333333254775" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "20771344.03356427" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "ds-idrepo-0" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "ds-cts-1" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "1.0847478128864947" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG "data": { 03:33:05 DEBUG { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG "data": { 03:33:05 DEBUG { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "result": [ 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG { 03:33:05 DEBUG }, 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "value": [ 03:33:05 DEBUG "pod": "ds-idrepo-1" 03:33:05 DEBUG 03:33:05 DEBUG }, 03:33:05 DEBUG 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "ds-idrepo-0" 03:33:05 DEBUG 03:33:05 DEBUG }, 03:33:05 DEBUG 03:33:05 DEBUG "value": [ 03:33:05 DEBUG "pod": "ds-cts-1" 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG }, 03:33:05 DEBUG "0" 03:33:05 DEBUG "value": [ 03:33:05 DEBUG ] 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG } 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "instance": "10.142.0.107:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "instance": "10.142.0.77:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "30610.933333333334" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "instance": "10.142.0.107:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "instance": "10.142.0.77:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "26326.13333333333" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975585" 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "6246760448" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975585" 03:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975585" 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "223.07545679819185" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "instance": "10.142.0.107:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "instance": "10.142.0.77:9100" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "0.00033333333333350387" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:05 INFO [http_cmd]: http status code OK 03:33:05 DEBUG --- status code --- 03:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:33:05 DEBUG --- http response --- 03:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:33:05 DEBUG ----- output ----- 03:33:05 DEBUG { 03:33:05 DEBUG "status": "success", 03:33:05 DEBUG "data": { 03:33:05 DEBUG "resultType": "vector", 03:33:05 DEBUG "result": [ 03:33:05 DEBUG { 03:33:05 DEBUG "metric": { 03:33:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG [...] 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG 03:33:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:33:05 DEBUG }, 03:33:05 DEBUG "value": [ 03:33:05 DEBUG 1712975585, 03:33:05 DEBUG "2.02517042406875" 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG ] 03:33:05 DEBUG } 03:33:05 DEBUG } 03:33:20 INFO 03:33:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:33:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:33:20 INFO [loop_until]: OK (rc = 0) 03:33:20 DEBUG --- stdout --- 03:33:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 18m 12373Mi ds-cts-1 18m 13494Mi ds-cts-2 17m 13381Mi ds-idrepo-0 13m 5596Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 13m 5869Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 5Mi overseer-0-59d79dcbf-spbbl 993m 5393Mi 03:33:20 DEBUG --- stderr --- 03:33:20 DEBUG 03:33:27 INFO 03:33:27 INFO [loop_until]: kubectl --namespace=xlou top node 03:33:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:33:27 INFO [loop_until]: OK (rc = 0) 03:33:27 DEBUG --- stdout --- 03:33:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4744Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1869Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10305Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 120m 0% 3114Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 60m 0% 10217Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4372Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 10327Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 66m 0% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 14336Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 76m 0% 14421Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6723Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 75m 0% 13341Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6399Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 9263Mi 15% 03:33:27 DEBUG --- stderr --- 03:33:27 DEBUG 03:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975644" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 ERROR [http_cmd]: ERROR 03:34:05 ERROR --- status code --- 03:34:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:34:05 ERROR --- http response --- 03:34:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:34:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "instance": "10.142.0.107:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "instance": "10.142.0.77:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "723398556672" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "ds-cts-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "ds-idrepo-0" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:05 DEBUG "pod": "ds-cts-1" 03:34:05 DEBUG }, 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG "29.599999999999998" 03:34:05 DEBUG ] 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "instance": "10.142.0.107:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "instance": "10.142.0.77:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0.0008333333333292595" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0.024" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "429831.87763008336" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "ds-idrepo-0" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "ds-cts-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "instance": "10.142.0.107:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "instance": "10.142.0.77:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "30547.733333333334" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "1.028424376600876" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "ds-idrepo-0" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG ] 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG } 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG ] 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "ds-idrepo-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "ds-cts-1" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "0" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975645" 03:34:05 INFO [http_cmd]: http status code OK 03:34:05 DEBUG --- status code --- 03:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:05 DEBUG --- http response --- 03:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:05 DEBUG ----- output ----- 03:34:05 DEBUG { 03:34:05 DEBUG "status": "success", 03:34:05 DEBUG "data": { 03:34:05 DEBUG "resultType": "vector", 03:34:05 DEBUG "result": [ 03:34:05 DEBUG { 03:34:05 DEBUG "metric": { 03:34:05 DEBUG "instance": "10.142.0.107:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG [...] 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG 03:34:05 DEBUG "instance": "10.142.0.77:9100" 03:34:05 DEBUG }, 03:34:05 DEBUG "value": [ 03:34:05 DEBUG 1712975645, 03:34:05 DEBUG "24253" 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG ] 03:34:05 DEBUG } 03:34:05 DEBUG } 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975645" 03:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975645" 03:34:06 INFO [http_cmd]: http status code OK 03:34:06 DEBUG --- status code --- 03:34:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:06 DEBUG --- http response --- 03:34:06 INFO [http_cmd]: http status code OK 03:34:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:34:06 DEBUG --- status code --- 03:34:06 DEBUG ----- output ----- 03:34:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:06 DEBUG { 03:34:06 DEBUG --- http response --- 03:34:06 DEBUG "status": "success", 03:34:06 DEBUG "data": { 03:34:06 DEBUG "resultType": "vector", 03:34:06 DEBUG "result": [ 03:34:06 DEBUG { 03:34:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:34:06 DEBUG "metric": { 03:34:06 DEBUG ----- output ----- 03:34:06 DEBUG "pod": "am-6d96779b5d-zn69c" 03:34:06 DEBUG { 03:34:06 DEBUG }, 03:34:06 DEBUG "status": "success", 03:34:06 DEBUG "value": [ 03:34:06 DEBUG "data": { 03:34:06 DEBUG 03:34:06 DEBUG "resultType": "vector", 03:34:06 DEBUG 03:34:06 DEBUG "result": [ 03:34:06 DEBUG 03:34:06 DEBUG { 03:34:06 DEBUG [...] 03:34:06 DEBUG "metric": { 03:34:06 DEBUG 03:34:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:34:06 DEBUG 03:34:06 DEBUG }, 03:34:06 DEBUG 03:34:06 DEBUG "value": [ 03:34:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:06 DEBUG 03:34:06 DEBUG }, 03:34:06 DEBUG 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 03:34:06 DEBUG 1712975645, 03:34:06 DEBUG [...] 03:34:06 DEBUG "0" 03:34:06 DEBUG 03:34:06 DEBUG ] 03:34:06 DEBUG 03:34:06 DEBUG } 03:34:06 DEBUG 03:34:06 DEBUG ] 03:34:06 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:34:06 DEBUG } 03:34:06 DEBUG }, 03:34:06 DEBUG } 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 1712975645, 03:34:06 DEBUG "216.2904290429043" 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG } 03:34:06 INFO [http_cmd]: http status code OK 03:34:06 DEBUG --- status code --- 03:34:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:06 DEBUG --- http response --- 03:34:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:06 DEBUG ----- output ----- 03:34:06 DEBUG { 03:34:06 DEBUG "status": "success", 03:34:06 DEBUG "data": { 03:34:06 DEBUG "resultType": "vector", 03:34:06 DEBUG "result": [ 03:34:06 DEBUG { 03:34:06 DEBUG "metric": { 03:34:06 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:34:06 DEBUG }, 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG [...] 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:34:06 DEBUG }, 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 1712975645, 03:34:06 DEBUG "6247112704" 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG } 03:34:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975646" 03:34:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975646" 03:34:06 INFO [http_cmd]: http status code OK 03:34:06 DEBUG --- status code --- 03:34:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:06 DEBUG --- http response --- 03:34:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:34:06 DEBUG ----- output ----- 03:34:06 DEBUG { 03:34:06 DEBUG "status": "success", 03:34:06 DEBUG "data": { 03:34:06 DEBUG "resultType": "vector", 03:34:06 DEBUG "result": [ 03:34:06 DEBUG { 03:34:06 DEBUG "metric": { 03:34:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:34:06 DEBUG }, 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG [...] 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:34:06 DEBUG }, 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 1712975646, 03:34:06 DEBUG "0.0038907683456870005" 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG } 03:34:06 INFO [http_cmd]: http status code OK 03:34:06 DEBUG --- status code --- 03:34:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:34:06 DEBUG --- http response --- 03:34:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:34:06 DEBUG ----- output ----- 03:34:06 DEBUG { 03:34:06 DEBUG "status": "success", 03:34:06 DEBUG "data": { 03:34:06 DEBUG "resultType": "vector", 03:34:06 DEBUG "result": [ 03:34:06 DEBUG { 03:34:06 DEBUG "metric": { 03:34:06 DEBUG "instance": "10.142.0.107:9100" 03:34:06 DEBUG }, 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG [...] 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG 03:34:06 DEBUG "instance": "10.142.0.77:9100" 03:34:06 DEBUG }, 03:34:06 DEBUG "value": [ 03:34:06 DEBUG 1712975646, 03:34:06 DEBUG "0" 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG ] 03:34:06 DEBUG } 03:34:06 DEBUG } 03:34:20 INFO 03:34:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:34:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:34:20 INFO [loop_until]: OK (rc = 0) 03:34:20 DEBUG --- stdout --- 03:34:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 8m 8481Mi am-6d96779b5d-qbg7l 6m 8416Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 17m 12373Mi ds-cts-1 17m 13494Mi ds-cts-2 18m 13381Mi ds-idrepo-0 13m 5595Mi ds-idrepo-1 12m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 5m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 1008m 5399Mi 03:34:20 DEBUG --- stderr --- 03:34:20 DEBUG 03:34:27 INFO 03:34:27 INFO [loop_until]: kubectl --namespace=xlou top node 03:34:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:34:27 INFO [loop_until]: OK (rc = 0) 03:34:27 DEBUG --- stdout --- 03:34:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4744Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 1867Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10302Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 113m 0% 3125Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 10218Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4347Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 10307Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 66m 0% 6315Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 68m 0% 14337Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 78m 0% 14425Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 69m 0% 6719Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 13340Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6400Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 9266Mi 15% 03:34:27 DEBUG --- stderr --- 03:34:27 DEBUG 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 ERROR [http_cmd]: ERROR 03:35:05 ERROR --- status code --- 03:35:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:35:05 ERROR --- http response --- 03:35:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:35:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-cts-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "instance": "10.142.0.107:9100" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "instance": "10.142.0.77:9100" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "723400084480" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-idrepo-0" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-cts-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "28.4" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:05 DEBUG { 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG { 03:35:05 DEBUG "data": { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "result": [ 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG { 03:35:05 DEBUG "result": [ 03:35:05 DEBUG "metric": { 03:35:05 DEBUG { 03:35:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:05 DEBUG "metric": { 03:35:05 DEBUG }, 03:35:05 DEBUG "instance": "10.142.0.107:9100" 03:35:05 DEBUG "value": [ 03:35:05 DEBUG }, 03:35:05 DEBUG 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG 03:35:05 DEBUG }, 03:35:05 DEBUG "instance": "10.142.0.77:9100" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG "value": [ 03:35:05 DEBUG }, 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG "0" 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG ] 03:35:05 DEBUG "0.0006999999999957633" 03:35:05 DEBUG } 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ] 03:35:05 DEBUG ] 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 DEBUG { 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG } 03:35:05 DEBUG "data": { 03:35:05 DEBUG } 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0.024" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "4802.353276476304" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-idrepo-0" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-cts-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:35:05 DEBUG } 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG } 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-cts-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "instance": "10.142.0.107:9100" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "instance": "10.142.0.77:9100" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "30523.466666666664" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975705" 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "1.0355270428673105" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "ds-idrepo-1" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "ds-idrepo-0" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "0" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975705" 03:35:05 INFO [http_cmd]: http status code OK 03:35:05 DEBUG --- status code --- 03:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:05 DEBUG --- http response --- 03:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:05 DEBUG ----- output ----- 03:35:05 DEBUG { 03:35:05 DEBUG "status": "success", 03:35:05 DEBUG "data": { 03:35:05 DEBUG "resultType": "vector", 03:35:05 DEBUG "result": [ 03:35:05 DEBUG { 03:35:05 DEBUG "metric": { 03:35:05 DEBUG "instance": "10.142.0.107:9100" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG [...] 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG 03:35:05 DEBUG "instance": "10.142.0.77:9100" 03:35:05 DEBUG }, 03:35:05 DEBUG "value": [ 03:35:05 DEBUG 1712975705, 03:35:05 DEBUG "23707.1" 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG ] 03:35:05 DEBUG } 03:35:05 DEBUG } 03:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975706" 03:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975706" 03:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975706" 03:35:06 INFO [http_cmd]: http status code OK 03:35:06 DEBUG --- status code --- 03:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:06 DEBUG --- http response --- 03:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:35:06 DEBUG ----- output ----- 03:35:06 DEBUG { 03:35:06 DEBUG "status": "success", 03:35:06 DEBUG "data": { 03:35:06 DEBUG "resultType": "vector", 03:35:06 DEBUG "result": [ 03:35:06 DEBUG { 03:35:06 DEBUG "metric": { 03:35:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG [...] 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 1712975706, 03:35:06 DEBUG "0.0029632289754464754" 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG } 03:35:06 INFO [http_cmd]: http status code OK 03:35:06 DEBUG --- status code --- 03:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:06 DEBUG --- http response --- 03:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:35:06 DEBUG ----- output ----- 03:35:06 DEBUG { 03:35:06 DEBUG "status": "success", 03:35:06 DEBUG "data": { 03:35:06 DEBUG "resultType": "vector", 03:35:06 DEBUG "result": [ 03:35:06 DEBUG { 03:35:06 DEBUG "metric": { 03:35:06 DEBUG "pod": "am-6d96779b5d-zn69c" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG [...] 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 1712975706, 03:35:06 DEBUG "0" 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG } 03:35:06 INFO [http_cmd]: http status code OK 03:35:06 DEBUG --- status code --- 03:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:06 DEBUG --- http response --- 03:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:35:06 DEBUG ----- output ----- 03:35:06 DEBUG { 03:35:06 DEBUG "status": "success", 03:35:06 DEBUG "data": { 03:35:06 DEBUG "resultType": "vector", 03:35:06 DEBUG "result": [ 03:35:06 DEBUG { 03:35:06 DEBUG "metric": { 03:35:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG [...] 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 1712975706, 03:35:06 DEBUG "0" 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG } 03:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975706" 03:35:06 INFO [http_cmd]: http status code OK 03:35:06 DEBUG --- status code --- 03:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:06 DEBUG --- http response --- 03:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:06 DEBUG ----- output ----- 03:35:06 DEBUG { 03:35:06 DEBUG "status": "success", 03:35:06 DEBUG "data": { 03:35:06 DEBUG "resultType": "vector", 03:35:06 DEBUG "result": [ 03:35:06 DEBUG { 03:35:06 DEBUG "metric": { 03:35:06 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG [...] 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 1712975706, 03:35:06 DEBUG "6246895616" 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG } 03:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975706" 03:35:06 INFO [http_cmd]: http status code OK 03:35:06 DEBUG --- status code --- 03:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:35:06 DEBUG --- http response --- 03:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:35:06 DEBUG ----- output ----- 03:35:06 DEBUG { 03:35:06 DEBUG "status": "success", 03:35:06 DEBUG "data": { 03:35:06 DEBUG "resultType": "vector", 03:35:06 DEBUG "result": [ 03:35:06 DEBUG { 03:35:06 DEBUG "metric": { 03:35:06 DEBUG "instance": "10.142.0.107:9100" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG [...] 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG 03:35:06 DEBUG "instance": "10.142.0.77:9100" 03:35:06 DEBUG }, 03:35:06 DEBUG "value": [ 03:35:06 DEBUG 1712975706, 03:35:06 DEBUG "0" 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG ] 03:35:06 DEBUG } 03:35:06 DEBUG } 03:35:20 INFO 03:35:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:35:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:35:20 INFO [loop_until]: OK (rc = 0) 03:35:20 DEBUG --- stdout --- 03:35:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 7m 8413Mi ds-cts-0 19m 12373Mi ds-cts-1 18m 12538Mi ds-cts-2 16m 13381Mi ds-idrepo-0 13m 5596Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 13m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 980m 5581Mi 03:35:20 DEBUG --- stderr --- 03:35:20 DEBUG 03:35:27 INFO 03:35:27 INFO [loop_until]: kubectl --namespace=xlou top node 03:35:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:35:27 INFO [loop_until]: OK (rc = 0) 03:35:27 DEBUG --- stdout --- 03:35:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 4752Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 1850Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10300Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 115m 0% 3126Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 10223Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 130m 0% 4378Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 10319Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 67m 0% 14336Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 80m 0% 13471Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 72m 0% 6724Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 13343Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 71m 0% 6400Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1070m 6% 9507Mi 16% 03:35:27 DEBUG --- stderr --- 03:35:27 DEBUG 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-cts-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 ERROR [http_cmd]: ERROR 03:36:05 ERROR --- status code --- 03:36:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:36:05 ERROR --- http response --- 03:36:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:36:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "instance": "10.142.0.107:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "instance": "10.142.0.77:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "723401604096" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-idrepo-0" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-cts-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "instance": "10.142.0.107:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "instance": "10.142.0.77:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0.0008666666666779102" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0.024" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-idrepo-0" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-cts-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-cts-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "8723.269607530052" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "instance": "10.142.0.107:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "instance": "10.142.0.77:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "30726.566666666666" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975765" 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0.9350464324529086" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "ds-idrepo-1" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "ds-idrepo-0" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "instance": "10.142.0.107:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "instance": "10.142.0.77:9100" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "24561.833333333336" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975765" 03:36:05 INFO [http_cmd]: http status code OK 03:36:05 DEBUG --- status code --- 03:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:05 DEBUG --- http response --- 03:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:05 DEBUG ----- output ----- 03:36:05 DEBUG { 03:36:05 DEBUG "status": "success", 03:36:05 DEBUG "data": { 03:36:05 DEBUG "resultType": "vector", 03:36:05 DEBUG "result": [ 03:36:05 DEBUG { 03:36:05 DEBUG "metric": { 03:36:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG [...] 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG 03:36:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:05 DEBUG }, 03:36:05 DEBUG "value": [ 03:36:05 DEBUG 1712975765, 03:36:05 DEBUG "0" 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG ] 03:36:05 DEBUG } 03:36:05 DEBUG } 03:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975766" 03:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975766" 03:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975766" 03:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975766" 03:36:06 INFO [http_cmd]: http status code OK 03:36:06 DEBUG --- status code --- 03:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:06 DEBUG --- http response --- 03:36:06 INFO [http_cmd]: http status code OK 03:36:06 DEBUG --- status code --- 03:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:06 DEBUG --- http response --- 03:36:06 INFO [http_cmd]: http status code OK 03:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:06 DEBUG --- status code --- 03:36:06 DEBUG ----- output ----- 03:36:06 DEBUG ----- output ----- 03:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:06 DEBUG { 03:36:06 DEBUG { 03:36:06 DEBUG --- http response --- 03:36:06 DEBUG "status": "success", 03:36:06 DEBUG "status": "success", 03:36:06 DEBUG "data": { 03:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:36:06 DEBUG "data": { 03:36:06 DEBUG "resultType": "vector", 03:36:06 DEBUG ----- output ----- 03:36:06 DEBUG "resultType": "vector", 03:36:06 DEBUG "result": [ 03:36:06 DEBUG { 03:36:06 DEBUG "result": [ 03:36:06 DEBUG { 03:36:06 DEBUG "status": "success", 03:36:06 DEBUG { 03:36:06 DEBUG "metric": { 03:36:06 DEBUG "data": { 03:36:06 DEBUG "metric": { 03:36:06 DEBUG "pod": "am-6d96779b5d-zn69c" 03:36:06 DEBUG "resultType": "vector", 03:36:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:36:06 DEBUG }, 03:36:06 DEBUG "result": [ 03:36:06 DEBUG }, 03:36:06 DEBUG "value": [ 03:36:06 DEBUG { 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 03:36:06 DEBUG "metric": { 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG }, 03:36:06 DEBUG 03:36:06 DEBUG [...] 03:36:06 DEBUG "value": [ 03:36:06 DEBUG [...] 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:06 DEBUG [...] 03:36:06 DEBUG "pod": "overseer-0-59d79dcbf-spbbl" 03:36:06 DEBUG }, 03:36:06 DEBUG 03:36:06 DEBUG }, 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 1712975766, 03:36:06 DEBUG 03:36:06 DEBUG 1712975766, 03:36:06 DEBUG "0" 03:36:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:36:06 DEBUG "62032970.02259181" 03:36:06 DEBUG ] 03:36:06 DEBUG }, 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG "value": [ 03:36:06 DEBUG } 03:36:06 DEBUG ] 03:36:06 DEBUG 1712975766, 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG "0" 03:36:06 DEBUG } 03:36:06 DEBUG } 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG } 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG } 03:36:06 INFO [http_cmd]: http status code OK 03:36:06 DEBUG --- status code --- 03:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:06 DEBUG --- http response --- 03:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:06 DEBUG ----- output ----- 03:36:06 DEBUG { 03:36:06 DEBUG "status": "success", 03:36:06 DEBUG "data": { 03:36:06 DEBUG "resultType": "vector", 03:36:06 DEBUG "result": [ 03:36:06 DEBUG { 03:36:06 DEBUG "metric": { 03:36:06 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:36:06 DEBUG }, 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG [...] 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:36:06 DEBUG }, 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 1712975766, 03:36:06 DEBUG "6246150144" 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG } 03:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975766" 03:36:06 INFO [http_cmd]: http status code OK 03:36:06 DEBUG --- status code --- 03:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:36:06 DEBUG --- http response --- 03:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:36:06 DEBUG ----- output ----- 03:36:06 DEBUG { 03:36:06 DEBUG "status": "success", 03:36:06 DEBUG "data": { 03:36:06 DEBUG "resultType": "vector", 03:36:06 DEBUG "result": [ 03:36:06 DEBUG { 03:36:06 DEBUG "metric": { 03:36:06 DEBUG "instance": "10.142.0.107:9100" 03:36:06 DEBUG }, 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG [...] 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG 03:36:06 DEBUG "instance": "10.142.0.77:9100" 03:36:06 DEBUG }, 03:36:06 DEBUG "value": [ 03:36:06 DEBUG 1712975766, 03:36:06 DEBUG "0.0003333333333330302" 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG ] 03:36:06 DEBUG } 03:36:06 DEBUG } 03:36:20 INFO 03:36:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:36:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:36:20 INFO [loop_until]: OK (rc = 0) 03:36:20 DEBUG --- stdout --- 03:36:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 8m 8482Mi am-6d96779b5d-qbg7l 6m 8416Mi am-6d96779b5d-zn69c 7m 8413Mi ds-cts-0 14m 12373Mi ds-cts-1 15m 12538Mi ds-cts-2 14m 12685Mi ds-idrepo-0 15m 5596Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 4m 1121Mi lodemon-8dc49bd56-qr24k 1m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 964m 5769Mi 03:36:20 DEBUG --- stderr --- 03:36:20 DEBUG 03:36:27 INFO 03:36:27 INFO [loop_until]: kubectl --namespace=xlou top node 03:36:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:36:27 INFO [loop_until]: OK (rc = 0) 03:36:27 DEBUG --- stdout --- 03:36:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 4748Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 1860Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10295Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 3119Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 10209Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 10311Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 66m 0% 6318Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 67m 0% 13650Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 74m 0% 13468Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 70m 0% 6721Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 69m 0% 13344Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 68m 0% 6400Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1095m 6% 9639Mi 16% 03:36:27 DEBUG --- stderr --- 03:36:27 DEBUG 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-cts-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975825" 03:37:05 ERROR [http_cmd]: ERROR 03:37:05 ERROR --- status code --- 03:37:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:37:05 ERROR --- http response --- 03:37:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:37:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "instance": "10.142.0.107:9100" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "instance": "10.142.0.77:9100" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "723402853376" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-idrepo-0" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG { 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-cts-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "instance": "10.142.0.107:9100" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "instance": "10.142.0.77:9100" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0.00046666666665515775" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0.024" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "160317761.80095908" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-idrepo-0" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-cts-1" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:37:05 DEBUG --- status code --- 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG { 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:05 DEBUG "data": { 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG { 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "result": [ 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG { 03:37:05 DEBUG "data": { 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG "result": [ 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG "result": [ 03:37:05 DEBUG }, 03:37:05 DEBUG { 03:37:05 DEBUG "result": [ 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "value": [ 03:37:05 DEBUG "metric": { 03:37:05 DEBUG { 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG 03:37:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "instance": "10.142.0.107:9100" 03:37:05 DEBUG 03:37:05 DEBUG }, 03:37:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:37:05 DEBUG "data": { 03:37:05 DEBUG }, 03:37:05 DEBUG 03:37:05 DEBUG "value": [ 03:37:05 DEBUG }, 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "value": [ 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG "value": [ 03:37:05 DEBUG "result": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG { 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "metric": { 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG [...] 03:37:05 DEBUG "pod": "ds-cts-1" 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG }, 03:37:05 DEBUG 03:37:05 DEBUG }, 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "pod": "ds-idrepo-1" 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "instance": "10.142.0.77:9100" 03:37:05 DEBUG "0" 03:37:05 DEBUG }, 03:37:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:37:05 DEBUG 03:37:05 DEBUG }, 03:37:05 DEBUG ] 03:37:05 DEBUG "value": [ 03:37:05 DEBUG }, 03:37:05 DEBUG [...] 03:37:05 DEBUG "value": [ 03:37:05 DEBUG } 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG ] 03:37:05 DEBUG "2153.539598000204" 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG 03:37:05 DEBUG "30590.2" 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG "1.0871416417280773" 03:37:05 DEBUG 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG "pod": "ds-idrepo-0" 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG }, 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG "value": [ 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975825" 03:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975825" 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "instance": "10.142.0.107:9100" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "instance": "10.142.0.77:9100" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "24113.9" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:05 INFO [http_cmd]: http status code OK 03:37:05 DEBUG --- status code --- 03:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:05 DEBUG --- http response --- 03:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:05 DEBUG ----- output ----- 03:37:05 DEBUG { 03:37:05 DEBUG "status": "success", 03:37:05 DEBUG "data": { 03:37:05 DEBUG "resultType": "vector", 03:37:05 DEBUG "result": [ 03:37:05 DEBUG { 03:37:05 DEBUG "metric": { 03:37:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG [...] 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG 03:37:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:05 DEBUG }, 03:37:05 DEBUG "value": [ 03:37:05 DEBUG 1712975825, 03:37:05 DEBUG "0" 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG ] 03:37:05 DEBUG } 03:37:05 DEBUG } 03:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975826" 03:37:06 INFO [http_cmd]: http status code OK 03:37:06 DEBUG --- status code --- 03:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:06 DEBUG --- http response --- 03:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:37:06 DEBUG ----- output ----- 03:37:06 DEBUG { 03:37:06 DEBUG "status": "success", 03:37:06 DEBUG "data": { 03:37:06 DEBUG "resultType": "vector", 03:37:06 DEBUG "result": [ 03:37:06 DEBUG { 03:37:06 DEBUG "metric": { 03:37:06 DEBUG "pod": "am-6d96779b5d-zn69c" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG [...] 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 1712975826, 03:37:06 DEBUG "0" 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG } 03:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975826" 03:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975826" 03:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975826" 03:37:06 INFO [http_cmd]: http status code OK 03:37:06 INFO [http_cmd]: http status code OK 03:37:06 DEBUG --- status code --- 03:37:06 DEBUG --- status code --- 03:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:06 DEBUG --- http response --- 03:37:06 DEBUG --- http response --- 03:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:37:06 DEBUG ----- output ----- 03:37:06 DEBUG { 03:37:06 DEBUG "status": "success", 03:37:06 DEBUG "data": { 03:37:06 DEBUG "resultType": "vector", 03:37:06 DEBUG "result": [ 03:37:06 DEBUG { 03:37:06 DEBUG "metric": { 03:37:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG [...] 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG "pod": "ds-idrepo-1" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 1712975826, 03:37:06 DEBUG "0" 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG } 03:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:06 DEBUG ----- output ----- 03:37:06 DEBUG { 03:37:06 DEBUG "status": "success", 03:37:06 DEBUG "data": { 03:37:06 DEBUG "resultType": "vector", 03:37:06 DEBUG "result": [ 03:37:06 DEBUG { 03:37:06 DEBUG "metric": { 03:37:06 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG [...] 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 1712975826, 03:37:06 DEBUG "6245543936" 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG } 03:37:06 INFO [http_cmd]: http status code OK 03:37:06 DEBUG --- status code --- 03:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:06 DEBUG --- http response --- 03:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:37:06 DEBUG ----- output ----- 03:37:06 DEBUG { 03:37:06 DEBUG "status": "success", 03:37:06 DEBUG "data": { 03:37:06 DEBUG "resultType": "vector", 03:37:06 DEBUG "result": [ 03:37:06 DEBUG { 03:37:06 DEBUG "metric": { 03:37:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG [...] 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG "pod": "ds-idrepo-1" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 1712975826, 03:37:06 DEBUG "0.007807154693393884" 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG } 03:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975826" 03:37:06 INFO [http_cmd]: http status code OK 03:37:06 DEBUG --- status code --- 03:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:37:06 DEBUG --- http response --- 03:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:37:06 DEBUG ----- output ----- 03:37:06 DEBUG { 03:37:06 DEBUG "status": "success", 03:37:06 DEBUG "data": { 03:37:06 DEBUG "resultType": "vector", 03:37:06 DEBUG "result": [ 03:37:06 DEBUG { 03:37:06 DEBUG "metric": { 03:37:06 DEBUG "instance": "10.142.0.107:9100" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG [...] 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG 03:37:06 DEBUG "instance": "10.142.0.77:9100" 03:37:06 DEBUG }, 03:37:06 DEBUG "value": [ 03:37:06 DEBUG 1712975826, 03:37:06 DEBUG "0" 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG ] 03:37:06 DEBUG } 03:37:06 DEBUG } 03:37:20 INFO 03:37:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:37:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:37:20 INFO [loop_until]: OK (rc = 0) 03:37:20 DEBUG --- stdout --- 03:37:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 6m 8413Mi ds-cts-0 14m 12373Mi ds-cts-1 15m 12538Mi ds-cts-2 12m 12685Mi ds-idrepo-0 15m 5596Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 13m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 1m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 997m 5779Mi 03:37:20 DEBUG --- stderr --- 03:37:20 DEBUG 03:37:27 INFO 03:37:27 INFO [loop_until]: kubectl --namespace=xlou top node 03:37:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:37:27 INFO [loop_until]: OK (rc = 0) 03:37:27 DEBUG --- stdout --- 03:37:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 4745Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 1848Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10297Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 113m 0% 3122Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 60m 0% 10209Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4355Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 10310Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6316Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 68m 0% 13642Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 73m 0% 13466Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 68m 0% 6724Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 70m 0% 13344Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6403Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 9649Mi 16% 03:37:27 DEBUG --- stderr --- 03:37:27 DEBUG 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-cts-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 ERROR [http_cmd]: ERROR 03:38:05 ERROR --- status code --- 03:38:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:38:05 ERROR --- http response --- 03:38:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:38:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "instance": "10.142.0.107:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "instance": "10.142.0.77:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "723403277312" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-cts-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0.03333333333333333" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-idrepo-0" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "instance": "10.142.0.107:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "instance": "10.142.0.77:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0.0004666666666684212" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0.024" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "1004.4139283962728" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-cts-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-idrepo-0" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "instance": "10.142.0.107:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:38:05 DEBUG }, 03:38:05 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-idrepo-0" 03:38:05 DEBUG 03:38:05 DEBUG "pod": "ds-cts-1" 03:38:05 DEBUG 03:38:05 DEBUG "instance": "10.142.0.77:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "pod": "ds-idrepo-1" 03:38:05 DEBUG }, 03:38:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "30491.866666666665" 03:38:05 DEBUG ] 03:38:05 DEBUG "2901.6402375892326" 03:38:05 DEBUG ] 03:38:05 DEBUG "1.0322500524214444" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975885" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975885" 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "instance": "10.142.0.107:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "instance": "10.142.0.77:9100" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "24217.833333333332" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:05 INFO [http_cmd]: http status code OK 03:38:05 DEBUG --- status code --- 03:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:05 DEBUG --- http response --- 03:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:05 DEBUG ----- output ----- 03:38:05 DEBUG { 03:38:05 DEBUG "status": "success", 03:38:05 DEBUG "data": { 03:38:05 DEBUG "resultType": "vector", 03:38:05 DEBUG "result": [ 03:38:05 DEBUG { 03:38:05 DEBUG "metric": { 03:38:05 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG [...] 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG 03:38:05 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:05 DEBUG }, 03:38:05 DEBUG "value": [ 03:38:05 DEBUG 1712975885, 03:38:05 DEBUG "0" 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG ] 03:38:05 DEBUG } 03:38:05 DEBUG } 03:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975886" 03:38:06 INFO [http_cmd]: http status code OK 03:38:06 DEBUG --- status code --- 03:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:06 DEBUG --- http response --- 03:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:06 DEBUG ----- output ----- 03:38:06 DEBUG { 03:38:06 DEBUG "status": "success", 03:38:06 DEBUG "data": { 03:38:06 DEBUG "resultType": "vector", 03:38:06 DEBUG "result": [ 03:38:06 DEBUG { 03:38:06 DEBUG "metric": { 03:38:06 DEBUG "pod": "am-6d96779b5d-zn69c" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG [...] 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG "pod": "am-6d96779b5d-pfljh" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 1712975886, 03:38:06 DEBUG "0" 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG } 03:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975886" 03:38:06 INFO [http_cmd]: http status code OK 03:38:06 DEBUG --- status code --- 03:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:06 DEBUG --- http response --- 03:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:38:06 DEBUG ----- output ----- 03:38:06 DEBUG { 03:38:06 DEBUG "status": "success", 03:38:06 DEBUG "data": { 03:38:06 DEBUG "resultType": "vector", 03:38:06 DEBUG "result": [ 03:38:06 DEBUG { 03:38:06 DEBUG "metric": { 03:38:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG [...] 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG "pod": "ds-idrepo-1" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 1712975886, 03:38:06 DEBUG "0.010253056481888545" 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG } 03:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712975886" 03:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712975886" 03:38:06 INFO [http_cmd]: http status code OK 03:38:06 DEBUG --- status code --- 03:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:06 DEBUG --- http response --- 03:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:06 DEBUG ----- output ----- 03:38:06 DEBUG { 03:38:06 DEBUG "status": "success", 03:38:06 DEBUG "data": { 03:38:06 DEBUG "resultType": "vector", 03:38:06 DEBUG "result": [ 03:38:06 DEBUG { 03:38:06 DEBUG "metric": { 03:38:06 DEBUG "pod": "admin-ui-c4c7bf584-vlxw7" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG [...] 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG "pod": "ds-idrepo-1" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 1712975886, 03:38:06 DEBUG "0" 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG } 03:38:06 INFO [http_cmd]: http status code OK 03:38:06 DEBUG --- status code --- 03:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:06 DEBUG --- http response --- 03:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:06 DEBUG ----- output ----- 03:38:06 DEBUG { 03:38:06 DEBUG "status": "success", 03:38:06 DEBUG "data": { 03:38:06 DEBUG "resultType": "vector", 03:38:06 DEBUG "result": [ 03:38:06 DEBUG { 03:38:06 DEBUG "metric": { 03:38:06 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-mpfw" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG [...] 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 1712975886, 03:38:06 DEBUG "6246543360" 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG } 03:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712975886" 03:38:06 INFO [http_cmd]: http status code OK 03:38:06 DEBUG --- status code --- 03:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:06 DEBUG --- http response --- 03:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:06 DEBUG ----- output ----- 03:38:06 DEBUG { 03:38:06 DEBUG "status": "success", 03:38:06 DEBUG "data": { 03:38:06 DEBUG "resultType": "vector", 03:38:06 DEBUG "result": [ 03:38:06 DEBUG { 03:38:06 DEBUG "metric": { 03:38:06 DEBUG "instance": "10.142.0.107:9100" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG [...] 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG 03:38:06 DEBUG "instance": "10.142.0.77:9100" 03:38:06 DEBUG }, 03:38:06 DEBUG "value": [ 03:38:06 DEBUG 1712975886, 03:38:06 DEBUG "0" 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG ] 03:38:06 DEBUG } 03:38:06 DEBUG } 03:38:20 INFO 03:38:20 INFO [loop_until]: kubectl --namespace=xlou top pods 03:38:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:20 INFO [loop_until]: OK (rc = 0) 03:38:20 DEBUG --- stdout --- 03:38:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-vlxw7 1m 9Mi am-6d96779b5d-pfljh 7m 8481Mi am-6d96779b5d-qbg7l 6m 8415Mi am-6d96779b5d-zn69c 7m 8413Mi ds-cts-0 13m 12373Mi ds-cts-1 15m 12537Mi ds-cts-2 14m 12685Mi ds-idrepo-0 14m 5596Mi ds-idrepo-1 11m 5523Mi ds-idrepo-2 14m 5868Mi end-user-ui-58878d7668-5gqgr 1m 5Mi idm-b85f9d978-b559t 3m 1121Mi idm-b85f9d978-dz57p 3m 1121Mi lodemon-8dc49bd56-qr24k 4m 69Mi login-ui-5fdf49fc66-svvhv 1m 4Mi overseer-0-59d79dcbf-spbbl 1032m 5779Mi 03:38:20 DEBUG --- stderr --- 03:38:20 DEBUG 03:38:27 INFO 03:38:27 INFO [loop_until]: kubectl --namespace=xlou top node 03:38:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:27 INFO [loop_until]: OK (rc = 0) 03:38:27 DEBUG --- stdout --- 03:38:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 4742Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 1853Mi 1% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10286Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 115m 0% 3108Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 61m 0% 10205Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 4371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 10304Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6318Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 66m 0% 13642Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 74m 0% 13470Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 71m 0% 6723Mi 11% gke-xlou-cdm-ds-32e4dcb1-j2hk 69m 0% 13344Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6396Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1092m 6% 9649Mi 16% 03:38:27 DEBUG --- stderr --- 03:38:27 DEBUG 03:38:59 INFO Finished: True 03:38:59 INFO Waiting for threads to register finish flag 03:39:27 INFO Done. Have a nice day! :) 127.0.0.1 - - [13/Apr/2024 03:39:27] "GET /monitoring/stop HTTP/1.1" 200 - 03:39:31 WARNING File /tmp/lodemon_data-20240412_220143/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6511', ...) 127.0.0.1 - - [13/Apr/2024 03:39:35] "GET /monitoring/process HTTP/1.1" 200 -