==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-df4b7b554-2gxfx Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Fri, 19 Apr 2024 21:06:39 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=df4b7b554 skaffold.dev/run-id=ca272a82-2f94-49c9-92a6-0e1265d22dbd Annotations: Status: Running IP: 10.106.42.188 IPs: IP: 10.106.42.188 Controlled By: ReplicaSet/lodemon-df4b7b554 Containers: lodemon: Container ID: containerd://df664ca460d9dadbf55bc2c8a1372f63aa336aba0f954a1f73cfb80d66f144a7 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 19 Apr 2024 21:06:40 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wl55x (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-wl55x: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 04:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713582733" 04:12:13 INFO [http_cmd]: http status code OK 04:12:13 DEBUG --- status code --- 04:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:13 DEBUG --- http response --- 04:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:13 DEBUG ----- output ----- 04:12:13 DEBUG { 04:12:13 DEBUG "status": "success", 04:12:13 DEBUG "data": { 04:12:13 DEBUG "resultType": "vector", 04:12:13 DEBUG "result": [ 04:12:13 DEBUG { 04:12:13 DEBUG "metric": { 04:12:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:12:13 DEBUG }, 04:12:13 DEBUG "value": [ 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG [...] 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:12:13 DEBUG }, 04:12:13 DEBUG "value": [ 04:12:13 DEBUG 1713582733, 04:12:13 DEBUG "1489383424" 04:12:13 DEBUG ] 04:12:13 DEBUG } 04:12:13 DEBUG ] 04:12:13 DEBUG } 04:12:13 DEBUG } 04:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713582733" 04:12:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582733" 04:12:13 INFO [http_cmd]: http status code OK 04:12:13 DEBUG --- status code --- 04:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:13 DEBUG --- http response --- 04:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:13 DEBUG ----- output ----- 04:12:13 DEBUG { 04:12:13 DEBUG "status": "success", 04:12:13 DEBUG "data": { 04:12:13 DEBUG "resultType": "vector", 04:12:13 DEBUG "result": [ 04:12:13 DEBUG { 04:12:13 DEBUG "metric": { 04:12:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:12:13 DEBUG }, 04:12:13 DEBUG "value": [ 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG [...] 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:12:13 DEBUG }, 04:12:13 DEBUG "value": [ 04:12:13 DEBUG 1713582733, 04:12:13 DEBUG "8.035304316151402" 04:12:13 DEBUG ] 04:12:13 DEBUG } 04:12:13 DEBUG ] 04:12:13 DEBUG } 04:12:13 DEBUG } 04:12:13 INFO [http_cmd]: http status code OK 04:12:13 DEBUG --- status code --- 04:12:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:13 DEBUG --- http response --- 04:12:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:13 DEBUG ----- output ----- 04:12:13 DEBUG { 04:12:13 DEBUG "status": "success", 04:12:13 DEBUG "data": { 04:12:13 DEBUG "resultType": "vector", 04:12:13 DEBUG "result": [ 04:12:13 DEBUG { 04:12:13 DEBUG "metric": { 04:12:13 DEBUG "instance": "10.142.0.107:9100" 04:12:13 DEBUG }, 04:12:13 DEBUG "value": [ 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG [...] 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG 04:12:13 DEBUG "instance": "10.142.0.43:9100" 04:12:13 DEBUG }, 04:12:13 DEBUG "value": [ 04:12:13 DEBUG 1713582733, 04:12:13 DEBUG "0.06433333333332598" 04:12:13 DEBUG ] 04:12:13 DEBUG } 04:12:13 DEBUG ] 04:12:13 DEBUG } 04:12:13 DEBUG } 04:12:29 INFO 04:12:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:12:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:12:29 INFO [loop_until]: OK (rc = 0) 04:12:29 DEBUG --- stdout --- 04:12:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17169m 8662Mi am-767f6b67b5-qcqz2 17431m 8731Mi am-767f6b67b5-vbltz 18806m 8658Mi ds-cts-0 19289m 10626Mi ds-cts-1 17638m 10722Mi ds-cts-2 19536m 10653Mi ds-idrepo-0 7512m 3690Mi ds-idrepo-1 8019m 3593Mi ds-idrepo-2 7999m 3519Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8090m 1327Mi 04:12:29 DEBUG --- stderr --- 04:12:29 DEBUG 04:12:33 INFO 04:12:33 INFO [loop_until]: kubectl --namespace=xlou top node 04:12:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:12:33 INFO [loop_until]: OK (rc = 0) 04:12:33 DEBUG --- stdout --- 04:12:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6267Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7828m 26% 7346Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17402m 58% 11372Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3339Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18974m 63% 13758Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5200Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17605m 58% 11611Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18869m 63% 2985Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19331m 64% 10868Mi 9% gke-xlou-cdm-ds-0f601751-j488 8229m 27% 6831Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17190m 57% 15625Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7983m 26% 7475Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19384m 64% 13214Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8198m 51% 4935Mi 8% 04:12:33 DEBUG --- stderr --- 04:12:33 DEBUG 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713582792" 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 ERROR [http_cmd]: ERROR 04:13:12 ERROR --- status code --- 04:13:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:13:12 ERROR --- http response --- 04:13:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:13:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG { 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG "data": { 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "pod": "ds-idrepo-0" 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG }, 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "value": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:12 DEBUG 04:13:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:12 DEBUG }, 04:13:12 DEBUG 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG [...] 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG "pod": "ds-idrepo-2" 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG }, 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG 04:13:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:12 DEBUG "0" 04:13:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:12 DEBUG }, 04:13:12 DEBUG ] 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG } 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG ] 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "2659.2" 04:13:12 DEBUG } 04:13:12 DEBUG "2653.5" 04:13:12 DEBUG ] 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "480099.5" 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "0" 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "ds-cts-0" 04:13:12 DEBUG "pod": "ds-idrepo-0" 04:13:12 DEBUG }, 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "ds-cts-2" 04:13:12 DEBUG "pod": "ds-idrepo-2" 04:13:12 DEBUG }, 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "23.03333333333333" 04:13:12 DEBUG "0" 04:13:12 DEBUG ] 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "2657.766666666667" 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "ds-idrepo-0" 04:13:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:13:12 DEBUG }, 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "ds-idrepo-2" 04:13:12 DEBUG "pod": "ds-cts-1" 04:13:12 DEBUG }, 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "0" 04:13:12 DEBUG "25991878.452620294" 04:13:12 DEBUG ] 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "2659.3333333333335" 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "pod": "ds-idrepo-2" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "0" 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582792" 04:13:12 INFO [http_cmd]: http status code OK 04:13:12 DEBUG --- status code --- 04:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:12 DEBUG --- http response --- 04:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:12 DEBUG ----- output ----- 04:13:12 DEBUG { 04:13:12 DEBUG "status": "success", 04:13:12 DEBUG "data": { 04:13:12 DEBUG "resultType": "vector", 04:13:12 DEBUG "result": [ 04:13:12 DEBUG { 04:13:12 DEBUG "metric": { 04:13:12 DEBUG "instance": "10.142.0.107:9100" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG [...] 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG 04:13:12 DEBUG "instance": "10.142.0.43:9100" 04:13:12 DEBUG }, 04:13:12 DEBUG "value": [ 04:13:12 DEBUG 1713582792, 04:13:12 DEBUG "80209949.43333334" 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG ] 04:13:12 DEBUG } 04:13:12 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582793" 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0.082" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "instance": "10.142.0.107:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "instance": "10.142.0.43:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "80210612.46666667" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "instance": "10.142.0.107:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "instance": "10.142.0.43:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0.16230000000001896" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "instance": "10.142.0.107:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "instance": "10.142.0.43:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "2992790410240" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582793" 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "pod": "ds-cts-0" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "pod": "ds-cts-2" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0.0003666666666666667" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "pod": "ds-idrepo-0" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "pod": "ds-cts-2" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582793" 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582793" 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "pod": "ds-cts-0" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "pod": "ds-cts-2" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "pod": "ds-cts-2" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "988.4766214177979" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "1491222528" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "8.03892397658539" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582793" 04:13:13 INFO [http_cmd]: http status code OK 04:13:13 DEBUG --- status code --- 04:13:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:13 DEBUG --- http response --- 04:13:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:13 DEBUG ----- output ----- 04:13:13 DEBUG { 04:13:13 DEBUG "status": "success", 04:13:13 DEBUG "data": { 04:13:13 DEBUG "resultType": "vector", 04:13:13 DEBUG "result": [ 04:13:13 DEBUG { 04:13:13 DEBUG "metric": { 04:13:13 DEBUG "instance": "10.142.0.107:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG [...] 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG 04:13:13 DEBUG "instance": "10.142.0.43:9100" 04:13:13 DEBUG }, 04:13:13 DEBUG "value": [ 04:13:13 DEBUG 1713582793, 04:13:13 DEBUG "0.062333333333342955" 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG ] 04:13:13 DEBUG } 04:13:13 DEBUG } 04:13:29 INFO 04:13:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:13:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:13:29 INFO [loop_until]: OK (rc = 0) 04:13:29 DEBUG --- stdout --- 04:13:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17679m 8664Mi am-767f6b67b5-qcqz2 17587m 8731Mi am-767f6b67b5-vbltz 18718m 8655Mi ds-cts-0 19158m 10606Mi ds-cts-1 17161m 10713Mi ds-cts-2 19669m 10647Mi ds-idrepo-0 7599m 3690Mi ds-idrepo-1 7652m 3592Mi ds-idrepo-2 7790m 3519Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 6m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8078m 1326Mi 04:13:29 DEBUG --- stderr --- 04:13:29 DEBUG 04:13:33 INFO 04:13:33 INFO [loop_until]: kubectl --namespace=xlou top node 04:13:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:13:34 INFO [loop_until]: OK (rc = 0) 04:13:34 DEBUG --- stdout --- 04:13:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6271Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7799m 26% 7346Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17421m 58% 11375Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3345Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19310m 64% 13743Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5178Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17561m 58% 11617Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18201m 60% 2989Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19340m 64% 10867Mi 9% gke-xlou-cdm-ds-0f601751-j488 8218m 27% 6836Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17522m 58% 15608Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7998m 26% 7473Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19234m 64% 13236Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 7930m 49% 4936Mi 8% 04:13:34 DEBUG --- stderr --- 04:13:34 DEBUG 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "2666.7" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713582852" 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 ERROR [http_cmd]: ERROR 04:14:12 ERROR --- status code --- 04:14:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:14:12 ERROR --- http response --- 04:14:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:14:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "ds-idrepo-0" 04:14:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:12 DEBUG }, 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "ds-idrepo-2" 04:14:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:12 DEBUG }, 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "0" 04:14:12 DEBUG "2675.366666666667" 04:14:12 DEBUG ] 04:14:12 DEBUG ] 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG ] 04:14:12 DEBUG ] 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "479885.5" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "0" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "ds-idrepo-0" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "ds-idrepo-2" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "0" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "ds-cts-0" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "ds-cts-2" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "25.03333333333333" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "2666.5666666666666" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "2666.5666666666666" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "ds-idrepo-0" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "ds-idrepo-2" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "0" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "0" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582852" 04:14:12 INFO [http_cmd]: http status code OK 04:14:12 DEBUG --- status code --- 04:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:12 DEBUG --- http response --- 04:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:14:12 DEBUG ----- output ----- 04:14:12 DEBUG { 04:14:12 DEBUG "status": "success", 04:14:12 DEBUG "data": { 04:14:12 DEBUG "resultType": "vector", 04:14:12 DEBUG "result": [ 04:14:12 DEBUG { 04:14:12 DEBUG "metric": { 04:14:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG [...] 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG 04:14:12 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:14:12 DEBUG }, 04:14:12 DEBUG "value": [ 04:14:12 DEBUG 1713582852, 04:14:12 DEBUG "0" 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG ] 04:14:12 DEBUG } 04:14:12 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "instance": "10.142.0.107:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "instance": "10.142.0.43:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "80886878.1" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713582853" 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0.082" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "instance": "10.142.0.107:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "instance": "10.142.0.43:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "80889396.56666666" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "instance": "10.142.0.107:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "instance": "10.142.0.43:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0.2489333333333036" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582853" 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "instance": "10.142.0.107:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "instance": "10.142.0.43:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "2996939219968" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713582853" 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "pod": "ds-cts-0" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "pod": "ds-cts-2" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0.001" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "pod": "ds-idrepo-0" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "pod": "ds-cts-2" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582853" 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582853" 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG { 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "pod": "ds-cts-0" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "pod": "ds-cts-2" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "1490235392" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "8.032722725941234" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582853" 04:14:13 INFO [http_cmd]: http status code OK 04:14:13 DEBUG --- status code --- 04:14:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:13 DEBUG --- http response --- 04:14:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:13 DEBUG ----- output ----- 04:14:13 DEBUG { 04:14:13 DEBUG "status": "success", 04:14:13 DEBUG "data": { 04:14:13 DEBUG "resultType": "vector", 04:14:13 DEBUG "result": [ 04:14:13 DEBUG { 04:14:13 DEBUG "metric": { 04:14:13 DEBUG "instance": "10.142.0.107:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG [...] 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG 04:14:13 DEBUG "instance": "10.142.0.43:9100" 04:14:13 DEBUG }, 04:14:13 DEBUG "value": [ 04:14:13 DEBUG 1713582853, 04:14:13 DEBUG "0.09133333333333363" 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG ] 04:14:13 DEBUG } 04:14:13 DEBUG } 04:14:29 INFO 04:14:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:14:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:14:29 INFO [loop_until]: OK (rc = 0) 04:14:29 DEBUG --- stdout --- 04:14:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 18181m 8664Mi am-767f6b67b5-qcqz2 17497m 8730Mi am-767f6b67b5-vbltz 18509m 8657Mi ds-cts-0 18765m 10626Mi ds-cts-1 17345m 10717Mi ds-cts-2 19386m 10655Mi ds-idrepo-0 7691m 3690Mi ds-idrepo-1 7666m 3593Mi ds-idrepo-2 7809m 3523Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8197m 1326Mi 04:14:29 DEBUG --- stderr --- 04:14:29 DEBUG 04:14:34 INFO 04:14:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:14:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:14:34 INFO [loop_until]: OK (rc = 0) 04:14:34 DEBUG --- stdout --- 04:14:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6274Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7846m 26% 7359Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17412m 58% 11372Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19029m 63% 13747Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 174m 0% 5162Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17663m 59% 11619Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18897m 63% 2994Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19466m 65% 10864Mi 9% gke-xlou-cdm-ds-0f601751-j488 8198m 27% 6836Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17291m 57% 15614Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7978m 26% 7468Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19748m 66% 13243Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8186m 51% 4937Mi 8% 04:14:34 DEBUG --- stderr --- 04:14:34 DEBUG 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713582912" 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 ERROR [http_cmd]: ERROR 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 ERROR --- status code --- 04:15:12 DEBUG --- status code --- 04:15:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 ERROR --- http response --- 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:15:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "2663.0666666666666" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "2672.866666666667" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "ds-idrepo-0" 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:12 DEBUG }, 04:15:12 DEBUG "pod": "ds-idrepo-0" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "ds-idrepo-2" 04:15:12 DEBUG 04:15:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:12 DEBUG }, 04:15:12 DEBUG "pod": "ds-idrepo-2" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "0" 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "479845.5" 04:15:12 DEBUG ] 04:15:12 DEBUG "0" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "0" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "ds-cts-0" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "ds-cts-2" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "29.933333333333334" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "2663.4333333333334" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "ds-idrepo-0" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "ds-idrepo-2" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "0" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582912" 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "2662.9666666666667" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:12 INFO [http_cmd]: http status code OK 04:15:12 DEBUG --- status code --- 04:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:12 DEBUG --- http response --- 04:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:12 DEBUG ----- output ----- 04:15:12 DEBUG { 04:15:12 DEBUG "status": "success", 04:15:12 DEBUG "data": { 04:15:12 DEBUG "resultType": "vector", 04:15:12 DEBUG "result": [ 04:15:12 DEBUG { 04:15:12 DEBUG "metric": { 04:15:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG [...] 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG 04:15:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:15:12 DEBUG }, 04:15:12 DEBUG "value": [ 04:15:12 DEBUG 1713582912, 04:15:12 DEBUG "0" 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG ] 04:15:12 DEBUG } 04:15:12 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "36296088.28315983" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "instance": "10.142.0.107:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "instance": "10.142.0.43:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "80584643.4" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0.082" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "instance": "10.142.0.107:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "instance": "10.142.0.43:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "80585550.53333333" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "instance": "10.142.0.107:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "instance": "10.142.0.43:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0.16526666666666376" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "instance": "10.142.0.107:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "instance": "10.142.0.43:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "3000841954304" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "ds-idrepo-0" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "ds-cts-2" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "ds-cts-0" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "ds-cts-2" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "ds-cts-0" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "ds-cts-2" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0.0014333333333333333" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0.0018111011678780629" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "1489772544" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "8.035850382157193" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582913" 04:15:13 INFO [http_cmd]: http status code OK 04:15:13 DEBUG --- status code --- 04:15:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:13 DEBUG --- http response --- 04:15:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:13 DEBUG ----- output ----- 04:15:13 DEBUG { 04:15:13 DEBUG "status": "success", 04:15:13 DEBUG "data": { 04:15:13 DEBUG "resultType": "vector", 04:15:13 DEBUG "result": [ 04:15:13 DEBUG { 04:15:13 DEBUG "metric": { 04:15:13 DEBUG "instance": "10.142.0.107:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG [...] 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG 04:15:13 DEBUG "instance": "10.142.0.43:9100" 04:15:13 DEBUG }, 04:15:13 DEBUG "value": [ 04:15:13 DEBUG 1713582913, 04:15:13 DEBUG "0.06266666666667978" 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG ] 04:15:13 DEBUG } 04:15:13 DEBUG } 04:15:29 INFO 04:15:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:15:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:15:29 INFO [loop_until]: OK (rc = 0) 04:15:29 DEBUG --- stdout --- 04:15:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17589m 8663Mi am-767f6b67b5-qcqz2 17640m 8729Mi am-767f6b67b5-vbltz 18735m 8657Mi ds-cts-0 19189m 10613Mi ds-cts-1 17211m 10710Mi ds-cts-2 18882m 10666Mi ds-idrepo-0 7696m 3691Mi ds-idrepo-1 7262m 3593Mi ds-idrepo-2 8003m 3523Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7869m 1327Mi 04:15:29 DEBUG --- stderr --- 04:15:29 DEBUG 04:15:34 INFO 04:15:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:15:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:15:34 INFO [loop_until]: OK (rc = 0) 04:15:34 DEBUG --- stdout --- 04:15:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6279Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7843m 26% 7354Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17390m 58% 11381Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19037m 63% 13750Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5177Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17599m 58% 11609Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18922m 63% 2990Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19412m 65% 10867Mi 9% gke-xlou-cdm-ds-0f601751-j488 8147m 27% 6840Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17153m 57% 15623Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7994m 26% 7475Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19105m 63% 13232Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8183m 51% 4937Mi 8% 04:15:34 DEBUG --- stderr --- 04:15:34 DEBUG 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713582972" 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "2659.7" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 ERROR [http_cmd]: ERROR 04:16:12 ERROR --- status code --- 04:16:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:16:12 ERROR --- http response --- 04:16:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:16:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "2658.4666666666667" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "ds-idrepo-0" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "ds-idrepo-2" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "0" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "0" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "479754" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "ds-idrepo-0" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "ds-idrepo-2" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "0" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "ds-cts-0" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "ds-cts-2" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "34.333333333333336" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "2657.233333333333" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582972" 04:16:12 INFO [http_cmd]: http status code OK 04:16:12 DEBUG --- status code --- 04:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:12 DEBUG --- http response --- 04:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:16:12 DEBUG ----- output ----- 04:16:12 DEBUG { 04:16:12 DEBUG "status": "success", 04:16:12 DEBUG "data": { 04:16:12 DEBUG "resultType": "vector", 04:16:12 DEBUG "result": [ 04:16:12 DEBUG { 04:16:12 DEBUG "metric": { 04:16:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG [...] 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG 04:16:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:16:12 DEBUG }, 04:16:12 DEBUG "value": [ 04:16:12 DEBUG 1713582972, 04:16:12 DEBUG "0" 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG ] 04:16:12 DEBUG } 04:16:12 DEBUG } 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582972, 04:16:13 DEBUG "2659.4333333333334" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "ds-idrepo-0" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "ds-idrepo-2" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "230.9266903254239" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582973" 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "instance": "10.142.0.107:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "instance": "10.142.0.43:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "80437053.96666667" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "instance": "10.142.0.107:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "instance": "10.142.0.43:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "80437015.73333333" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0.082" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "instance": "10.142.0.107:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "instance": "10.142.0.43:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0.27526666666668026" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582973" 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "instance": "10.142.0.107:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "instance": "10.142.0.43:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "3004607798272" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "ds-idrepo-0" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "ds-cts-2" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "ds-cts-0" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "ds-cts-2" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "ds-cts-0" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "ds-cts-2" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0.0009" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0.0013153406124594856" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "1490833408" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "8.041919260724574" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713582973" 04:16:13 INFO [http_cmd]: http status code OK 04:16:13 DEBUG --- status code --- 04:16:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:13 DEBUG --- http response --- 04:16:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:13 DEBUG ----- output ----- 04:16:13 DEBUG { 04:16:13 DEBUG "status": "success", 04:16:13 DEBUG "data": { 04:16:13 DEBUG "resultType": "vector", 04:16:13 DEBUG "result": [ 04:16:13 DEBUG { 04:16:13 DEBUG "metric": { 04:16:13 DEBUG "instance": "10.142.0.107:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG [...] 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG 04:16:13 DEBUG "instance": "10.142.0.43:9100" 04:16:13 DEBUG }, 04:16:13 DEBUG "value": [ 04:16:13 DEBUG 1713582973, 04:16:13 DEBUG "0.10299999999999349" 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG ] 04:16:13 DEBUG } 04:16:13 DEBUG } 04:16:29 INFO 04:16:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:16:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:16:29 INFO [loop_until]: OK (rc = 0) 04:16:29 DEBUG --- stdout --- 04:16:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17508m 8662Mi am-767f6b67b5-qcqz2 17995m 8731Mi am-767f6b67b5-vbltz 18502m 8656Mi ds-cts-0 19506m 10624Mi ds-cts-1 17727m 10711Mi ds-cts-2 19834m 10649Mi ds-idrepo-0 7674m 3695Mi ds-idrepo-1 7491m 3593Mi ds-idrepo-2 7912m 3523Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8105m 1326Mi 04:16:29 DEBUG --- stderr --- 04:16:29 DEBUG 04:16:34 INFO 04:16:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:16:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:16:34 INFO [loop_until]: OK (rc = 0) 04:16:34 DEBUG --- stdout --- 04:16:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6275Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7820m 26% 7355Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17400m 58% 11380Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19116m 64% 13755Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 172m 0% 5175Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 16929m 56% 11621Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18940m 63% 2991Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19408m 65% 10859Mi 9% gke-xlou-cdm-ds-0f601751-j488 7963m 26% 6832Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17355m 58% 15613Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7964m 26% 7468Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19575m 65% 13232Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8195m 51% 4939Mi 8% 04:16:34 DEBUG --- stderr --- 04:16:34 DEBUG 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583032" 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 ERROR [http_cmd]: ERROR 04:17:12 ERROR --- status code --- 04:17:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:17:12 ERROR --- http response --- 04:17:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:17:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "2672.366666666667" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "ds-cts-0" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "ds-cts-2" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "30" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "2669.7" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "479932" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "ds-idrepo-0" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "ds-idrepo-2" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "0" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "0" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:12 DEBUG ----- output ----- 04:17:12 DEBUG { 04:17:12 DEBUG "status": "success", 04:17:12 DEBUG "data": { 04:17:12 DEBUG "resultType": "vector", 04:17:12 DEBUG "result": [ 04:17:12 DEBUG { 04:17:12 DEBUG "metric": { 04:17:12 DEBUG "pod": "ds-idrepo-0" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG [...] 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG 04:17:12 DEBUG "pod": "ds-idrepo-2" 04:17:12 DEBUG }, 04:17:12 DEBUG "value": [ 04:17:12 DEBUG 1713583032, 04:17:12 DEBUG "0" 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG ] 04:17:12 DEBUG } 04:17:12 DEBUG } 04:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583032" 04:17:12 INFO [http_cmd]: http status code OK 04:17:12 DEBUG --- status code --- 04:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:12 DEBUG --- http response --- 04:17:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "2674.4666666666667" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "2671.9" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "ds-idrepo-0" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "ds-idrepo-2" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "53935.95126903553" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583033" 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "instance": "10.142.0.107:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "instance": "10.142.0.43:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "80741615.16666666" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "instance": "10.142.0.107:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "instance": "10.142.0.43:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "80740337.46666667" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0.082" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "instance": "10.142.0.107:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "instance": "10.142.0.43:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0.2505666666665926" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "instance": "10.142.0.107:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "instance": "10.142.0.43:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "3009139461120" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "ds-idrepo-0" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "ds-cts-2" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "ds-cts-0" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "ds-cts-2" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0.000033333333333333335" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "ds-cts-0" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "ds-cts-2" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "9.205861273023475" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "1491488768" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "8.038820051980752" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583033" 04:17:13 INFO [http_cmd]: http status code OK 04:17:13 DEBUG --- status code --- 04:17:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:13 DEBUG --- http response --- 04:17:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:13 DEBUG ----- output ----- 04:17:13 DEBUG { 04:17:13 DEBUG "status": "success", 04:17:13 DEBUG "data": { 04:17:13 DEBUG "resultType": "vector", 04:17:13 DEBUG "result": [ 04:17:13 DEBUG { 04:17:13 DEBUG "metric": { 04:17:13 DEBUG "instance": "10.142.0.107:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG [...] 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG 04:17:13 DEBUG "instance": "10.142.0.43:9100" 04:17:13 DEBUG }, 04:17:13 DEBUG "value": [ 04:17:13 DEBUG 1713583033, 04:17:13 DEBUG "0.10899999999999181" 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG ] 04:17:13 DEBUG } 04:17:13 DEBUG } 04:17:29 INFO 04:17:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:17:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:17:29 INFO [loop_until]: OK (rc = 0) 04:17:29 DEBUG --- stdout --- 04:17:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17566m 8664Mi am-767f6b67b5-qcqz2 18286m 8727Mi am-767f6b67b5-vbltz 18606m 8658Mi ds-cts-0 19237m 10627Mi ds-cts-1 17740m 10711Mi ds-cts-2 19172m 10664Mi ds-idrepo-0 7832m 3694Mi ds-idrepo-1 8034m 3593Mi ds-idrepo-2 7904m 3523Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7857m 1327Mi 04:17:29 DEBUG --- stderr --- 04:17:29 DEBUG 04:17:34 INFO 04:17:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:17:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:17:34 INFO [loop_until]: OK (rc = 0) 04:17:34 DEBUG --- stdout --- 04:17:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6271Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7839m 26% 7363Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17465m 58% 11378Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18818m 63% 13746Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 5198Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17539m 58% 11617Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18890m 63% 2996Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19383m 64% 10869Mi 9% gke-xlou-cdm-ds-0f601751-j488 8174m 27% 6833Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17734m 59% 15619Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7983m 26% 7473Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19210m 64% 13240Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8187m 51% 4935Mi 8% 04:17:34 DEBUG --- stderr --- 04:17:34 DEBUG 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 ERROR [http_cmd]: ERROR 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 ERROR --- status code --- 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG --- status code --- 04:18:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 ERROR --- http response --- 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "data": { 04:18:12 DEBUG "data": { 04:18:12 DEBUG "data": { 04:18:12 DEBUG "data": { 04:18:12 DEBUG "data": { 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "result": [ 04:18:12 DEBUG "result": [ 04:18:12 DEBUG "result": [ 04:18:12 DEBUG "result": [ 04:18:12 DEBUG "result": [ 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "pod": "ds-idrepo-0" 04:18:12 DEBUG "pod": "ds-idrepo-0" 04:18:12 DEBUG "pod": "ds-cts-0" 04:18:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG [...] 04:18:12 DEBUG [...] 04:18:12 DEBUG [...] 04:18:12 DEBUG [...] 04:18:12 DEBUG [...] 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583092" 04:18:12 DEBUG 04:18:12 DEBUG "pod": "ds-idrepo-2" 04:18:12 DEBUG "pod": "ds-idrepo-2" 04:18:12 DEBUG "pod": "ds-cts-2" 04:18:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG }, 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG "value": [ 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG "0" 04:18:12 DEBUG "0" 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 DEBUG "33.46666666666667" 04:18:12 DEBUG "2679.4333333333334" 04:18:12 DEBUG "0" 04:18:12 DEBUG ] 04:18:12 DEBUG ] 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG ] 04:18:12 DEBUG ] 04:18:12 DEBUG ] 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG ] 04:18:12 DEBUG ] 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG ] 04:18:12 DEBUG ] 04:18:12 DEBUG ] 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 DEBUG { 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "data": { 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "result": [ 04:18:12 DEBUG { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:12 DEBUG }, 04:18:12 DEBUG "value": [ 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG [...] 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:12 DEBUG }, 04:18:12 DEBUG "value": [ 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG "2673.1" 04:18:12 DEBUG ] 04:18:12 DEBUG } 04:18:12 DEBUG ] 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 INFO [http_cmd]: http status code OK 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG --- status code --- 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG --- http response --- 04:18:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:12 DEBUG ----- output ----- 04:18:12 DEBUG { 04:18:12 DEBUG "status": "success", 04:18:12 DEBUG "data": { 04:18:12 DEBUG "resultType": "vector", 04:18:12 DEBUG "result": [ 04:18:12 DEBUG { 04:18:12 DEBUG "metric": { 04:18:12 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:12 DEBUG }, 04:18:12 DEBUG "value": [ 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG [...] 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG 04:18:12 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:12 DEBUG }, 04:18:12 DEBUG "value": [ 04:18:12 DEBUG 1713583092, 04:18:12 DEBUG "480290.5" 04:18:12 DEBUG ] 04:18:12 DEBUG } 04:18:12 DEBUG ] 04:18:12 DEBUG } 04:18:12 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "2678.6666666666665" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "2676.366666666667" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "ds-idrepo-0" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "ds-idrepo-2" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "2382463.8293721755" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "instance": "10.142.0.107:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "instance": "10.142.0.43:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "81234664.93333334" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583093" 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "instance": "10.142.0.107:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "instance": "10.142.0.43:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "81235282.43333334" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0.082" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "instance": "10.142.0.107:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "instance": "10.142.0.43:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0.28503333333327513" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "instance": "10.142.0.107:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "instance": "10.142.0.43:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "3013326412800" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "ds-idrepo-0" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "ds-cts-2" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "ds-cts-0" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "ds-cts-2" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0.00046666666666666666" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "ds-cts-0" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "ds-cts-2" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "3995.1880872646875" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0.004890594011119319" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "1487015936" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "8.032679983434344" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583093" 04:18:13 INFO [http_cmd]: http status code OK 04:18:13 DEBUG --- status code --- 04:18:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:18:13 DEBUG --- http response --- 04:18:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:18:13 DEBUG ----- output ----- 04:18:13 DEBUG { 04:18:13 DEBUG "status": "success", 04:18:13 DEBUG "data": { 04:18:13 DEBUG "resultType": "vector", 04:18:13 DEBUG "result": [ 04:18:13 DEBUG { 04:18:13 DEBUG "metric": { 04:18:13 DEBUG "instance": "10.142.0.107:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG [...] 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG 04:18:13 DEBUG "instance": "10.142.0.43:9100" 04:18:13 DEBUG }, 04:18:13 DEBUG "value": [ 04:18:13 DEBUG 1713583093, 04:18:13 DEBUG "0.1036666666666823" 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG ] 04:18:13 DEBUG } 04:18:13 DEBUG } 04:18:29 INFO 04:18:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:18:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:18:29 INFO [loop_until]: OK (rc = 0) 04:18:29 DEBUG --- stdout --- 04:18:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 18032m 8663Mi am-767f6b67b5-qcqz2 17644m 8729Mi am-767f6b67b5-vbltz 19375m 8654Mi ds-cts-0 18636m 10629Mi ds-cts-1 17166m 10713Mi ds-cts-2 19022m 10661Mi ds-idrepo-0 7967m 3695Mi ds-idrepo-1 7595m 3609Mi ds-idrepo-2 7997m 3522Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7978m 1327Mi 04:18:29 DEBUG --- stderr --- 04:18:29 DEBUG 04:18:34 INFO 04:18:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:18:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:18:34 INFO [loop_until]: OK (rc = 0) 04:18:34 DEBUG --- stdout --- 04:18:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6263Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7859m 26% 7347Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17599m 58% 11373Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18906m 63% 13754Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17703m 59% 11623Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18976m 63% 2989Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19555m 65% 10863Mi 9% gke-xlou-cdm-ds-0f601751-j488 8215m 27% 6841Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17512m 58% 15602Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7983m 26% 7497Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19128m 64% 13241Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8177m 51% 4932Mi 8% 04:18:34 DEBUG --- stderr --- 04:18:34 DEBUG 04:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583152" 04:19:12 INFO [http_cmd]: http status code OK 04:19:12 DEBUG --- status code --- 04:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:12 DEBUG --- http response --- 04:19:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583152" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 ERROR [http_cmd]: ERROR 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 ERROR --- status code --- 04:19:13 DEBUG --- status code --- 04:19:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 ERROR --- http response --- 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:19:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "2672" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "ds-cts-0" 04:19:13 DEBUG "pod": "ds-idrepo-0" 04:19:13 DEBUG }, 04:19:13 DEBUG "pod": "ds-idrepo-0" 04:19:13 DEBUG }, 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG 04:19:13 DEBUG "pod": "ds-cts-2" 04:19:13 DEBUG "pod": "ds-idrepo-2" 04:19:13 DEBUG }, 04:19:13 DEBUG "pod": "ds-idrepo-2" 04:19:13 DEBUG }, 04:19:13 DEBUG }, 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG "value": [ 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG "value": [ 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG "2661.1666666666665" 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "30" 04:19:13 DEBUG "0" 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG ] 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG "0" 04:19:13 DEBUG ] 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ] 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG } 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG ] 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG { 04:19:13 DEBUG ] 04:19:13 DEBUG { 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG } 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG "data": { 04:19:13 DEBUG } 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG }, 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG }, 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0" 04:19:13 DEBUG "480971.5" 04:19:13 DEBUG ] 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "2672.1" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "2673.4666666666667" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "ds-idrepo-0" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "ds-idrepo-2" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "4821.6597998822845" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "instance": "10.142.0.107:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "instance": "10.142.0.43:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "80640339.73333333" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0.082" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "instance": "10.142.0.107:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "instance": "10.142.0.43:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "80643791.96666667" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "instance": "10.142.0.107:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "instance": "10.142.0.43:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0.1630999999998996" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "instance": "10.142.0.107:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "instance": "10.142.0.43:9100" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "3017419842560" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "567.2540941037981" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "ds-idrepo-0" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG "pod": "ds-cts-2" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0.003396115361976128" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "ds-cts-0" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "ds-cts-2" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0.00006666666666666672" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "pod": "ds-cts-0" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "pod": "ds-cts-2" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "0" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "1487716352" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583153" 04:19:13 INFO [http_cmd]: http status code OK 04:19:13 DEBUG --- status code --- 04:19:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:13 DEBUG --- http response --- 04:19:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:13 DEBUG ----- output ----- 04:19:13 DEBUG { 04:19:13 DEBUG "status": "success", 04:19:13 DEBUG "data": { 04:19:13 DEBUG "resultType": "vector", 04:19:13 DEBUG "result": [ 04:19:13 DEBUG { 04:19:13 DEBUG "metric": { 04:19:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG [...] 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG 04:19:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:19:13 DEBUG }, 04:19:13 DEBUG "value": [ 04:19:13 DEBUG 1713583153, 04:19:13 DEBUG "8.032855996964196" 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG ] 04:19:13 DEBUG } 04:19:13 DEBUG } 04:19:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583153" 04:19:14 INFO [http_cmd]: http status code OK 04:19:14 DEBUG --- status code --- 04:19:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:19:14 DEBUG --- http response --- 04:19:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:19:14 DEBUG ----- output ----- 04:19:14 DEBUG { 04:19:14 DEBUG "status": "success", 04:19:14 DEBUG "data": { 04:19:14 DEBUG "resultType": "vector", 04:19:14 DEBUG "result": [ 04:19:14 DEBUG { 04:19:14 DEBUG "metric": { 04:19:14 DEBUG "instance": "10.142.0.107:9100" 04:19:14 DEBUG }, 04:19:14 DEBUG "value": [ 04:19:14 DEBUG 04:19:14 DEBUG 04:19:14 DEBUG 04:19:14 DEBUG [...] 04:19:14 DEBUG 04:19:14 DEBUG 04:19:14 DEBUG 04:19:14 DEBUG "instance": "10.142.0.43:9100" 04:19:14 DEBUG }, 04:19:14 DEBUG "value": [ 04:19:14 DEBUG 1713583153, 04:19:14 DEBUG "0.06433333333332408" 04:19:14 DEBUG ] 04:19:14 DEBUG } 04:19:14 DEBUG ] 04:19:14 DEBUG } 04:19:14 DEBUG } 04:19:29 INFO 04:19:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:19:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:19:29 INFO [loop_until]: OK (rc = 0) 04:19:29 DEBUG --- stdout --- 04:19:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17820m 8661Mi am-767f6b67b5-qcqz2 17515m 8729Mi am-767f6b67b5-vbltz 18753m 8656Mi ds-cts-0 19156m 10628Mi ds-cts-1 17260m 10713Mi ds-cts-2 19840m 10650Mi ds-idrepo-0 7727m 3694Mi ds-idrepo-1 7582m 3609Mi ds-idrepo-2 7796m 3522Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 1m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7930m 1327Mi 04:19:29 DEBUG --- stderr --- 04:19:29 DEBUG 04:19:34 INFO 04:19:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:19:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:19:34 INFO [loop_until]: OK (rc = 0) 04:19:34 DEBUG --- stdout --- 04:19:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6282Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7828m 26% 7345Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17452m 58% 11374Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3333Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18426m 61% 13754Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5168Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17608m 58% 11617Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18648m 62% 2990Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19458m 65% 10872Mi 9% gke-xlou-cdm-ds-0f601751-j488 8192m 27% 6843Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17290m 57% 15626Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7891m 26% 7493Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19438m 65% 13241Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8204m 51% 4937Mi 8% 04:19:34 DEBUG --- stderr --- 04:19:34 DEBUG 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 ERROR [http_cmd]: ERROR 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 ERROR --- status code --- 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 ERROR --- http response --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG --- http response --- 04:20:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:20:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "ds-idrepo-0" 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-idrepo-2" 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG }, 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG "2666.9" 04:20:13 DEBUG ] 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "2668.233333333333" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG ----- output ----- 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG { 04:20:13 DEBUG { 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG "data": { 04:20:13 DEBUG "data": { 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "ds-idrepo-0" 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "pod": "ds-cts-0" 04:20:13 DEBUG }, 04:20:13 DEBUG "data": { 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG "result": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG { 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "metric": { 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG }, 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-idrepo-2" 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-cts-2" 04:20:13 DEBUG }, 04:20:13 DEBUG 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG [...] 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG 04:20:13 DEBUG "30" 04:20:13 DEBUG ] 04:20:13 DEBUG 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG }, 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG "value": [ 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG } 04:20:13 DEBUG "480762" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "2667.9" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "2666.7" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "ds-idrepo-0" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-idrepo-2" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "324918027.7579053" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "instance": "10.142.0.107:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "instance": "10.142.0.43:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "80999880.9" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0.082" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "instance": "10.142.0.107:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "instance": "10.142.0.43:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "81000983.66666667" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "instance": "10.142.0.107:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "instance": "10.142.0.43:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0.258533333333286" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "instance": "10.142.0.107:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "instance": "10.142.0.43:9100" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "3021764236288" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "ds-idrepo-0" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-cts-2" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "ds-cts-0" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-cts-2" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0.000733333333333333" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583213" 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "586.0638145657463" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "ds-cts-0" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "ds-cts-2" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "0" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "17.408250107311225" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583213" 04:20:13 INFO [http_cmd]: http status code OK 04:20:13 DEBUG --- status code --- 04:20:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:13 DEBUG --- http response --- 04:20:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:13 DEBUG ----- output ----- 04:20:13 DEBUG { 04:20:13 DEBUG "status": "success", 04:20:13 DEBUG "data": { 04:20:13 DEBUG "resultType": "vector", 04:20:13 DEBUG "result": [ 04:20:13 DEBUG { 04:20:13 DEBUG "metric": { 04:20:13 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG [...] 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG 04:20:13 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:20:13 DEBUG }, 04:20:13 DEBUG "value": [ 04:20:13 DEBUG 1713583213, 04:20:13 DEBUG "1488064512" 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG ] 04:20:13 DEBUG } 04:20:13 DEBUG } 04:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583214" 04:20:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583214" 04:20:14 INFO [http_cmd]: http status code OK 04:20:14 DEBUG --- status code --- 04:20:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:14 DEBUG --- http response --- 04:20:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:14 DEBUG ----- output ----- 04:20:14 DEBUG { 04:20:14 DEBUG "status": "success", 04:20:14 DEBUG "data": { 04:20:14 DEBUG "resultType": "vector", 04:20:14 DEBUG "result": [ 04:20:14 DEBUG { 04:20:14 DEBUG "metric": { 04:20:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:20:14 DEBUG }, 04:20:14 DEBUG "value": [ 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG [...] 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:20:14 DEBUG }, 04:20:14 DEBUG "value": [ 04:20:14 DEBUG 1713583214, 04:20:14 DEBUG "8.03835246079205" 04:20:14 DEBUG ] 04:20:14 DEBUG } 04:20:14 DEBUG ] 04:20:14 DEBUG } 04:20:14 DEBUG } 04:20:14 INFO [http_cmd]: http status code OK 04:20:14 DEBUG --- status code --- 04:20:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:20:14 DEBUG --- http response --- 04:20:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:20:14 DEBUG ----- output ----- 04:20:14 DEBUG { 04:20:14 DEBUG "status": "success", 04:20:14 DEBUG "data": { 04:20:14 DEBUG "resultType": "vector", 04:20:14 DEBUG "result": [ 04:20:14 DEBUG { 04:20:14 DEBUG "metric": { 04:20:14 DEBUG "instance": "10.142.0.107:9100" 04:20:14 DEBUG }, 04:20:14 DEBUG "value": [ 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG [...] 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG 04:20:14 DEBUG "instance": "10.142.0.43:9100" 04:20:14 DEBUG }, 04:20:14 DEBUG "value": [ 04:20:14 DEBUG 1713583214, 04:20:14 DEBUG "0.0986666666666622" 04:20:14 DEBUG ] 04:20:14 DEBUG } 04:20:14 DEBUG ] 04:20:14 DEBUG } 04:20:14 DEBUG } 04:20:29 INFO 04:20:29 INFO [loop_until]: kubectl --namespace=xlou top pods 04:20:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:20:30 INFO [loop_until]: OK (rc = 0) 04:20:30 DEBUG --- stdout --- 04:20:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17504m 8664Mi am-767f6b67b5-qcqz2 17470m 8730Mi am-767f6b67b5-vbltz 19184m 8656Mi ds-cts-0 19592m 10628Mi ds-cts-1 17742m 10718Mi ds-cts-2 19440m 10649Mi ds-idrepo-0 7544m 3695Mi ds-idrepo-1 7580m 3625Mi ds-idrepo-2 7915m 3522Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7940m 1327Mi 04:20:30 DEBUG --- stderr --- 04:20:30 DEBUG 04:20:34 INFO 04:20:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:20:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:20:34 INFO [loop_until]: OK (rc = 0) 04:20:34 DEBUG --- stdout --- 04:20:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6270Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7842m 26% 7354Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 16951m 56% 11374Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3340Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19024m 63% 13755Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5170Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17523m 58% 11617Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18885m 63% 2989Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19639m 65% 10867Mi 9% gke-xlou-cdm-ds-0f601751-j488 8197m 27% 6839Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17657m 59% 15599Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 8009m 26% 7477Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19499m 65% 13244Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8190m 51% 4938Mi 8% 04:20:34 DEBUG --- stderr --- 04:20:34 DEBUG 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG ----- output ----- 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG { 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG { 04:21:13 DEBUG { 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG "data": { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "data": { 04:21:13 DEBUG "data": { 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG "result": [ 04:21:13 DEBUG "result": [ 04:21:13 ERROR [http_cmd]: ERROR 04:21:13 DEBUG { 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG { 04:21:13 DEBUG { 04:21:13 ERROR --- status code --- 04:21:13 DEBUG "metric": { 04:21:13 DEBUG { 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "metric": { 04:21:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG "pod": "ds-idrepo-0" 04:21:13 DEBUG "pod": "ds-cts-0" 04:21:13 ERROR --- http response --- 04:21:13 DEBUG }, 04:21:13 DEBUG "data": { 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG }, 04:21:13 DEBUG }, 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG "value": [ 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:21:13 DEBUG "result": [ 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG { 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "metric": { 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG "pod": "ds-idrepo-0" 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG [...] 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG }, 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG "pod": "ds-idrepo-2" 04:21:13 DEBUG "pod": "ds-cts-2" 04:21:13 DEBUG }, 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG }, 04:21:13 DEBUG }, 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG [...] 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG "value": [ 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0" 04:21:13 DEBUG 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "2677.366666666667" 04:21:13 DEBUG "0" 04:21:13 DEBUG "30" 04:21:13 DEBUG ] 04:21:13 DEBUG 04:21:13 DEBUG "2655.4666666666667" 04:21:13 DEBUG ] 04:21:13 DEBUG ] 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG "pod": "ds-idrepo-2" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG }, 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG ] 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG "value": [ 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 DEBUG "0" 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "480699" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "2655.4333333333334" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "2656.1" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "ds-idrepo-0" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "ds-idrepo-2" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "ds-cts-2" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "1125.1201758000275" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG { 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:21:13 DEBUG "result": [ 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "instance": "10.142.0.107:9100" 04:21:13 DEBUG "data": { 04:21:13 DEBUG }, 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "value": [ 04:21:13 DEBUG "result": [ 04:21:13 DEBUG 04:21:13 DEBUG { 04:21:13 DEBUG 04:21:13 DEBUG "metric": { 04:21:13 DEBUG 04:21:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:21:13 DEBUG [...] 04:21:13 DEBUG }, 04:21:13 DEBUG 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "instance": "10.142.0.43:9100" 04:21:13 DEBUG 04:21:13 DEBUG }, 04:21:13 DEBUG [...] 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG 04:21:13 DEBUG "80716206.30000001" 04:21:13 DEBUG 04:21:13 DEBUG ] 04:21:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:21:13 DEBUG } 04:21:13 DEBUG }, 04:21:13 DEBUG ] 04:21:13 DEBUG "value": [ 04:21:13 DEBUG } 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG } 04:21:13 DEBUG "0.082" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "instance": "10.142.0.107:9100" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "instance": "10.142.0.43:9100" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "80717131.2" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "instance": "10.142.0.107:9100" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "instance": "10.142.0.43:9100" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0.27126666666668764" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "instance": "10.142.0.107:9100" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "instance": "10.142.0.43:9100" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "3026198406144" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "ds-idrepo-0" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "ds-cts-2" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "ds-cts-0" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "ds-cts-2" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0" 04:21:13 DEBUG ] 04:21:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583273" 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG ] 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG } 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG } 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "ds-cts-2" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "66867439.08803736" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:13 INFO [http_cmd]: http status code OK 04:21:13 DEBUG --- status code --- 04:21:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:13 DEBUG --- http response --- 04:21:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:21:13 DEBUG ----- output ----- 04:21:13 DEBUG { 04:21:13 DEBUG "status": "success", 04:21:13 DEBUG "data": { 04:21:13 DEBUG "resultType": "vector", 04:21:13 DEBUG "result": [ 04:21:13 DEBUG { 04:21:13 DEBUG "metric": { 04:21:13 DEBUG "pod": "ds-cts-0" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG [...] 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG 04:21:13 DEBUG "pod": "ds-cts-2" 04:21:13 DEBUG }, 04:21:13 DEBUG "value": [ 04:21:13 DEBUG 1713583273, 04:21:13 DEBUG "0.0004" 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG ] 04:21:13 DEBUG } 04:21:13 DEBUG } 04:21:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583274" 04:21:14 INFO [http_cmd]: http status code OK 04:21:14 DEBUG --- status code --- 04:21:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:14 DEBUG --- http response --- 04:21:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:14 DEBUG ----- output ----- 04:21:14 DEBUG { 04:21:14 DEBUG "status": "success", 04:21:14 DEBUG "data": { 04:21:14 DEBUG "resultType": "vector", 04:21:14 DEBUG "result": [ 04:21:14 DEBUG { 04:21:14 DEBUG "metric": { 04:21:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:21:14 DEBUG }, 04:21:14 DEBUG "value": [ 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG [...] 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:21:14 DEBUG }, 04:21:14 DEBUG "value": [ 04:21:14 DEBUG 1713583274, 04:21:14 DEBUG "1489027072" 04:21:14 DEBUG ] 04:21:14 DEBUG } 04:21:14 DEBUG ] 04:21:14 DEBUG } 04:21:14 DEBUG } 04:21:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583274" 04:21:14 INFO [http_cmd]: http status code OK 04:21:14 DEBUG --- status code --- 04:21:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:14 DEBUG --- http response --- 04:21:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:14 DEBUG ----- output ----- 04:21:14 DEBUG { 04:21:14 DEBUG "status": "success", 04:21:14 DEBUG "data": { 04:21:14 DEBUG "resultType": "vector", 04:21:14 DEBUG "result": [ 04:21:14 DEBUG { 04:21:14 DEBUG "metric": { 04:21:14 DEBUG "instance": "10.142.0.107:9100" 04:21:14 DEBUG }, 04:21:14 DEBUG "value": [ 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG [...] 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG "instance": "10.142.0.43:9100" 04:21:14 DEBUG }, 04:21:14 DEBUG "value": [ 04:21:14 DEBUG 1713583274, 04:21:14 DEBUG "0.10933333333333618" 04:21:14 DEBUG ] 04:21:14 DEBUG } 04:21:14 DEBUG ] 04:21:14 DEBUG } 04:21:14 DEBUG } 04:21:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583274" 04:21:14 INFO [http_cmd]: http status code OK 04:21:14 DEBUG --- status code --- 04:21:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:21:14 DEBUG --- http response --- 04:21:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:21:14 DEBUG ----- output ----- 04:21:14 DEBUG { 04:21:14 DEBUG "status": "success", 04:21:14 DEBUG "data": { 04:21:14 DEBUG "resultType": "vector", 04:21:14 DEBUG "result": [ 04:21:14 DEBUG { 04:21:14 DEBUG "metric": { 04:21:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:21:14 DEBUG }, 04:21:14 DEBUG "value": [ 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG [...] 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG 04:21:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:21:14 DEBUG }, 04:21:14 DEBUG "value": [ 04:21:14 DEBUG 1713583274, 04:21:14 DEBUG "8.033289385948237" 04:21:14 DEBUG ] 04:21:14 DEBUG } 04:21:14 DEBUG ] 04:21:14 DEBUG } 04:21:14 DEBUG } 04:21:30 INFO 04:21:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:21:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:21:30 INFO [loop_until]: OK (rc = 0) 04:21:30 DEBUG --- stdout --- 04:21:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17555m 8664Mi am-767f6b67b5-qcqz2 17579m 8728Mi am-767f6b67b5-vbltz 18800m 8657Mi ds-cts-0 19508m 10631Mi ds-cts-1 18092m 10712Mi ds-cts-2 19728m 10649Mi ds-idrepo-0 7633m 3695Mi ds-idrepo-1 7849m 3625Mi ds-idrepo-2 7880m 3523Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8071m 1327Mi 04:21:30 DEBUG --- stderr --- 04:21:30 DEBUG 04:21:34 INFO 04:21:34 INFO [loop_until]: kubectl --namespace=xlou top node 04:21:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:21:35 INFO [loop_until]: OK (rc = 0) 04:21:35 DEBUG --- stdout --- 04:21:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6276Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7848m 26% 7350Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17545m 58% 11376Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3329Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18864m 63% 13750Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5178Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17548m 58% 11618Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18980m 63% 2997Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19540m 65% 10875Mi 9% gke-xlou-cdm-ds-0f601751-j488 8167m 27% 6838Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17505m 58% 15613Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7995m 26% 7512Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19634m 65% 13238Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8181m 51% 4938Mi 8% 04:21:35 DEBUG --- stderr --- 04:21:35 DEBUG 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "ds-idrepo-0" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "ds-idrepo-2" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 ERROR [http_cmd]: ERROR 04:22:13 ERROR --- status code --- 04:22:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:22:13 ERROR --- http response --- 04:22:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:22:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:22:13 DEBUG { 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG "result": [ 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG { 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG { 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG "metric": { 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG "metric": { 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG }, 04:22:13 DEBUG }, 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG "value": [ 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:22:13 DEBUG "value": [ 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG 04:22:13 DEBUG { 04:22:13 DEBUG { 04:22:13 DEBUG 04:22:13 DEBUG { 04:22:13 DEBUG 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG 04:22:13 DEBUG "data": { 04:22:13 DEBUG "data": { 04:22:13 DEBUG [...] 04:22:13 DEBUG "data": { 04:22:13 DEBUG [...] 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG 04:22:13 DEBUG "result": [ 04:22:13 DEBUG "result": [ 04:22:13 DEBUG 04:22:13 DEBUG "result": [ 04:22:13 DEBUG 04:22:13 DEBUG { 04:22:13 DEBUG { 04:22:13 DEBUG 04:22:13 DEBUG { 04:22:13 DEBUG 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG "pod": "ds-idrepo-0" 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG }, 04:22:13 DEBUG "pod": "ds-cts-0" 04:22:13 DEBUG }, 04:22:13 DEBUG }, 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG "value": [ 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "2669.733333333333" 04:22:13 DEBUG 04:22:13 DEBUG "479988.5" 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG ] 04:22:13 DEBUG 04:22:13 DEBUG ] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG } 04:22:13 DEBUG 04:22:13 DEBUG } 04:22:13 DEBUG [...] 04:22:13 DEBUG [...] 04:22:13 DEBUG ] 04:22:13 DEBUG [...] 04:22:13 DEBUG ] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG } 04:22:13 DEBUG 04:22:13 DEBUG } 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG } 04:22:13 DEBUG 04:22:13 DEBUG } 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "ds-idrepo-2" 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG "pod": "ds-cts-2" 04:22:13 DEBUG }, 04:22:13 DEBUG }, 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG "value": [ 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG "2664.9666666666667" 04:22:13 DEBUG "29.966666666666665" 04:22:13 DEBUG ] 04:22:13 DEBUG ] 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG ] 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "2664.4333333333334" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "2667.5" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "ds-idrepo-0" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "ds-idrepo-2" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "8218.96849035899" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0.082" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "instance": "10.142.0.107:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "instance": "10.142.0.43:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "80650502.36666666" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "instance": "10.142.0.107:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "instance": "10.142.0.43:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "80652329.03333333" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "instance": "10.142.0.107:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "instance": "10.142.0.43:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0.22270000000001552" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "instance": "10.142.0.107:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "instance": "10.142.0.43:9100" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "3030190535680" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "ds-idrepo-0" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "ds-cts-2" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0.0028845587082607605" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583333" 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "513.6855306474368" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "ds-cts-0" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "ds-cts-2" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0.0013666666666666664" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:13 INFO [http_cmd]: http status code OK 04:22:13 DEBUG --- status code --- 04:22:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:13 DEBUG --- http response --- 04:22:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:22:13 DEBUG ----- output ----- 04:22:13 DEBUG { 04:22:13 DEBUG "status": "success", 04:22:13 DEBUG "data": { 04:22:13 DEBUG "resultType": "vector", 04:22:13 DEBUG "result": [ 04:22:13 DEBUG { 04:22:13 DEBUG "metric": { 04:22:13 DEBUG "pod": "ds-cts-0" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG [...] 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG 04:22:13 DEBUG "pod": "ds-cts-2" 04:22:13 DEBUG }, 04:22:13 DEBUG "value": [ 04:22:13 DEBUG 1713583333, 04:22:13 DEBUG "0" 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG ] 04:22:13 DEBUG } 04:22:13 DEBUG } 04:22:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583334" 04:22:14 INFO [http_cmd]: http status code OK 04:22:14 DEBUG --- status code --- 04:22:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:14 DEBUG --- http response --- 04:22:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:14 DEBUG ----- output ----- 04:22:14 DEBUG { 04:22:14 DEBUG "status": "success", 04:22:14 DEBUG "data": { 04:22:14 DEBUG "resultType": "vector", 04:22:14 DEBUG "result": [ 04:22:14 DEBUG { 04:22:14 DEBUG "metric": { 04:22:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:22:14 DEBUG }, 04:22:14 DEBUG "value": [ 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG [...] 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:22:14 DEBUG }, 04:22:14 DEBUG "value": [ 04:22:14 DEBUG 1713583334, 04:22:14 DEBUG "1491238912" 04:22:14 DEBUG ] 04:22:14 DEBUG } 04:22:14 DEBUG ] 04:22:14 DEBUG } 04:22:14 DEBUG } 04:22:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583334" 04:22:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583334" 04:22:14 INFO [http_cmd]: http status code OK 04:22:14 DEBUG --- status code --- 04:22:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:14 DEBUG --- http response --- 04:22:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:14 DEBUG ----- output ----- 04:22:14 DEBUG { 04:22:14 DEBUG "status": "success", 04:22:14 DEBUG "data": { 04:22:14 DEBUG "resultType": "vector", 04:22:14 INFO [http_cmd]: http status code OK 04:22:14 DEBUG "result": [ 04:22:14 DEBUG --- status code --- 04:22:14 DEBUG { 04:22:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:22:14 DEBUG "metric": { 04:22:14 DEBUG --- http response --- 04:22:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:22:14 DEBUG }, 04:22:14 DEBUG "value": [ 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG [...] 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:22:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:22:14 DEBUG ----- output ----- 04:22:14 DEBUG }, 04:22:14 DEBUG { 04:22:14 DEBUG "value": [ 04:22:14 DEBUG "status": "success", 04:22:14 DEBUG 1713583334, 04:22:14 DEBUG "data": { 04:22:14 DEBUG "8.034340658237179" 04:22:14 DEBUG "resultType": "vector", 04:22:14 DEBUG ] 04:22:14 DEBUG "result": [ 04:22:14 DEBUG } 04:22:14 DEBUG { 04:22:14 DEBUG ] 04:22:14 DEBUG "metric": { 04:22:14 DEBUG } 04:22:14 DEBUG "instance": "10.142.0.107:9100" 04:22:14 DEBUG } 04:22:14 DEBUG }, 04:22:14 DEBUG "value": [ 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG [...] 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG 04:22:14 DEBUG "instance": "10.142.0.43:9100" 04:22:14 DEBUG }, 04:22:14 DEBUG "value": [ 04:22:14 DEBUG 1713583334, 04:22:14 DEBUG "0.08466666666666543" 04:22:14 DEBUG ] 04:22:14 DEBUG } 04:22:14 DEBUG ] 04:22:14 DEBUG } 04:22:14 DEBUG } 04:22:30 INFO 04:22:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:22:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:22:30 INFO [loop_until]: OK (rc = 0) 04:22:30 DEBUG --- stdout --- 04:22:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17185m 8666Mi am-767f6b67b5-qcqz2 17625m 8732Mi am-767f6b67b5-vbltz 19046m 8656Mi ds-cts-0 19456m 10630Mi ds-cts-1 17657m 10713Mi ds-cts-2 19261m 10660Mi ds-idrepo-0 7598m 3694Mi ds-idrepo-1 7813m 3625Mi ds-idrepo-2 7993m 3522Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8063m 1327Mi 04:22:30 DEBUG --- stderr --- 04:22:30 DEBUG 04:22:35 INFO 04:22:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:22:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:22:35 INFO [loop_until]: OK (rc = 0) 04:22:35 DEBUG --- stdout --- 04:22:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6273Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7813m 26% 7350Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17477m 58% 11383Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3338Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18945m 63% 13752Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5176Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17580m 58% 11618Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18699m 62% 2987Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19345m 64% 10870Mi 9% gke-xlou-cdm-ds-0f601751-j488 8147m 27% 6824Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17393m 58% 15613Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 7658m 25% 7505Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19296m 64% 13245Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8206m 51% 4936Mi 8% 04:22:35 DEBUG --- stderr --- 04:22:35 DEBUG 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG "data": { 04:23:13 DEBUG { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG "data": { 04:23:13 DEBUG { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "result": [ 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG { 04:23:13 DEBUG }, 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "pod": "ds-idrepo-0" 04:23:13 DEBUG 04:23:13 DEBUG }, 04:23:13 DEBUG 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG 04:23:13 DEBUG }, 04:23:13 DEBUG 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "pod": "ds-idrepo-2" 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG }, 04:23:13 DEBUG "0" 04:23:13 DEBUG "value": [ 04:23:13 DEBUG ] 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG } 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "451840.5" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "2674.3333333333335" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "ds-cts-0" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "ds-cts-2" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "30.933333333333334" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:23:13 ERROR [http_cmd]: ERROR 04:23:13 ERROR --- status code --- 04:23:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:23:13 ERROR --- http response --- 04:23:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:23:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "ds-idrepo-0" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "ds-idrepo-2" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "790.5333333333333" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "791.4666666666667" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "789.2666666666667" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "ds-idrepo-0" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "ds-idrepo-2" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:23:13 DEBUG "instance": "10.142.0.107:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:23:13 DEBUG "instance": "10.142.0.43:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0.082" 04:23:13 DEBUG "17556182.666666668" 04:23:13 DEBUG ] 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "instance": "10.142.0.107:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "instance": "10.142.0.43:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "17554480.333333336" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "instance": "10.142.0.107:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "instance": "10.142.0.43:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "3033104270336" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "instance": "10.142.0.107:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "instance": "10.142.0.43:9100" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0.3266000000000152" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "ds-idrepo-0" 04:23:13 DEBUG "pod": "ds-cts-0" 04:23:13 DEBUG }, 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG { 04:23:13 DEBUG "pod": "ds-cts-2" 04:23:13 DEBUG "pod": "ds-cts-2" 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG }, 04:23:13 DEBUG }, 04:23:13 DEBUG "data": { 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "value": [ 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "result": [ 04:23:13 DEBUG "0" 04:23:13 DEBUG "0" 04:23:13 DEBUG { 04:23:13 DEBUG ] 04:23:13 DEBUG ] 04:23:13 DEBUG "metric": { 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:23:13 DEBUG ] 04:23:13 DEBUG ] 04:23:13 DEBUG }, 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG "value": [ 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583393" 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "598101.1748245" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:13 INFO [http_cmd]: http status code OK 04:23:13 DEBUG --- status code --- 04:23:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:13 DEBUG --- http response --- 04:23:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:23:13 DEBUG ----- output ----- 04:23:13 DEBUG { 04:23:13 DEBUG "status": "success", 04:23:13 DEBUG "data": { 04:23:13 DEBUG "resultType": "vector", 04:23:13 DEBUG "result": [ 04:23:13 DEBUG { 04:23:13 DEBUG "metric": { 04:23:13 DEBUG "pod": "ds-cts-0" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG [...] 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG 04:23:13 DEBUG "pod": "ds-cts-2" 04:23:13 DEBUG }, 04:23:13 DEBUG "value": [ 04:23:13 DEBUG 1713583393, 04:23:13 DEBUG "0" 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG ] 04:23:13 DEBUG } 04:23:13 DEBUG } 04:23:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583394" 04:23:14 INFO [http_cmd]: http status code OK 04:23:14 DEBUG --- status code --- 04:23:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:14 DEBUG --- http response --- 04:23:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:14 DEBUG ----- output ----- 04:23:14 DEBUG { 04:23:14 DEBUG "status": "success", 04:23:14 DEBUG "data": { 04:23:14 DEBUG "resultType": "vector", 04:23:14 DEBUG "result": [ 04:23:14 DEBUG { 04:23:14 DEBUG "metric": { 04:23:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:23:14 DEBUG }, 04:23:14 DEBUG "value": [ 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG [...] 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:23:14 DEBUG }, 04:23:14 DEBUG "value": [ 04:23:14 DEBUG 1713583394, 04:23:14 DEBUG "653594624" 04:23:14 DEBUG ] 04:23:14 DEBUG } 04:23:14 DEBUG ] 04:23:14 DEBUG } 04:23:14 DEBUG } 04:23:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583394" 04:23:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583394" 04:23:14 INFO [http_cmd]: http status code OK 04:23:14 DEBUG --- status code --- 04:23:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:14 DEBUG --- http response --- 04:23:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:14 DEBUG ----- output ----- 04:23:14 DEBUG { 04:23:14 DEBUG "status": "success", 04:23:14 DEBUG "data": { 04:23:14 DEBUG "resultType": "vector", 04:23:14 DEBUG "result": [ 04:23:14 DEBUG { 04:23:14 DEBUG "metric": { 04:23:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:23:14 DEBUG }, 04:23:14 DEBUG "value": [ 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG [...] 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:23:14 DEBUG }, 04:23:14 DEBUG "value": [ 04:23:14 DEBUG 1713583394, 04:23:14 DEBUG "7.5267055597056345" 04:23:14 DEBUG ] 04:23:14 DEBUG } 04:23:14 DEBUG ] 04:23:14 DEBUG } 04:23:14 DEBUG } 04:23:14 INFO [http_cmd]: http status code OK 04:23:14 DEBUG --- status code --- 04:23:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:23:14 DEBUG --- http response --- 04:23:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:23:14 DEBUG ----- output ----- 04:23:14 DEBUG { 04:23:14 DEBUG "status": "success", 04:23:14 DEBUG "data": { 04:23:14 DEBUG "resultType": "vector", 04:23:14 DEBUG "result": [ 04:23:14 DEBUG { 04:23:14 DEBUG "metric": { 04:23:14 DEBUG "instance": "10.142.0.107:9100" 04:23:14 DEBUG }, 04:23:14 DEBUG "value": [ 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG [...] 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG 04:23:14 DEBUG "instance": "10.142.0.43:9100" 04:23:14 DEBUG }, 04:23:14 DEBUG "value": [ 04:23:14 DEBUG 1713583394, 04:23:14 DEBUG "0.1789999999999945" 04:23:14 DEBUG ] 04:23:14 DEBUG } 04:23:14 DEBUG ] 04:23:14 DEBUG } 04:23:14 DEBUG } 04:23:30 INFO 04:23:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:23:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:23:30 INFO [loop_until]: OK (rc = 0) 04:23:30 DEBUG --- stdout --- 04:23:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2515m 8658Mi am-767f6b67b5-qcqz2 2601m 8727Mi am-767f6b67b5-vbltz 2981m 8654Mi ds-cts-0 8122m 10675Mi ds-cts-1 7573m 10707Mi ds-cts-2 8252m 10622Mi ds-idrepo-0 771m 3694Mi ds-idrepo-1 758m 3625Mi ds-idrepo-2 736m 3523Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 526Mi 04:23:30 DEBUG --- stderr --- 04:23:30 DEBUG 04:23:35 INFO 04:23:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:23:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:23:35 INFO [loop_until]: OK (rc = 0) 04:23:35 DEBUG --- stdout --- 04:23:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6276Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 824m 2% 7360Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2554m 8% 11408Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 77m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8037m 26% 13811Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 171m 0% 5196Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2589m 8% 11643Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2970Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3129m 10% 10916Mi 9% gke-xlou-cdm-ds-0f601751-j488 846m 2% 6817Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 7371m 24% 15625Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 856m 2% 7518Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8341m 27% 13245Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 71m 0% 4148Mi 7% 04:23:35 DEBUG --- stderr --- 04:23:35 DEBUG 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "303742" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-cts-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-cts-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "30" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-idrepo-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-idrepo-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "2655.1" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 ERROR [http_cmd]: ERROR 04:24:13 ERROR --- status code --- 04:24:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:24:13 ERROR --- http response --- 04:24:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:24:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-idrepo-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-idrepo-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG }, 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG }, 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-idrepo-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-idrepo-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-idrepo-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0.082" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "instance": "10.142.0.107:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "instance": "10.142.0.43:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "17489126.799999997" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "instance": "10.142.0.107:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "instance": "10.142.0.43:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "17489097.46666667" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "instance": "10.142.0.107:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "instance": "10.142.0.43:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "3034866889728" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "instance": "10.142.0.107:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "instance": "10.142.0.43:9100" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0.09033333333335729" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-cts-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-cts-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0.00003333333333333336" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583453" 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG { 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-idrepo-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-cts-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-idrepo-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:13 INFO [http_cmd]: http status code OK 04:24:13 DEBUG --- status code --- 04:24:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:13 DEBUG --- http response --- 04:24:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:24:13 DEBUG ----- output ----- 04:24:13 DEBUG { 04:24:13 DEBUG "status": "success", 04:24:13 DEBUG "data": { 04:24:13 DEBUG "resultType": "vector", 04:24:13 DEBUG "result": [ 04:24:13 DEBUG { 04:24:13 DEBUG "metric": { 04:24:13 DEBUG "pod": "ds-cts-0" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG [...] 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG 04:24:13 DEBUG "pod": "ds-cts-2" 04:24:13 DEBUG }, 04:24:13 DEBUG "value": [ 04:24:13 DEBUG 1713583453, 04:24:13 DEBUG "0" 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG ] 04:24:13 DEBUG } 04:24:13 DEBUG } 04:24:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583454" 04:24:14 INFO [http_cmd]: http status code OK 04:24:14 DEBUG --- status code --- 04:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:14 DEBUG --- http response --- 04:24:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:14 DEBUG ----- output ----- 04:24:14 DEBUG { 04:24:14 DEBUG "status": "success", 04:24:14 DEBUG "data": { 04:24:14 DEBUG "resultType": "vector", 04:24:14 DEBUG "result": [ 04:24:14 DEBUG { 04:24:14 DEBUG "metric": { 04:24:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:24:14 DEBUG }, 04:24:14 DEBUG "value": [ 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG [...] 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:24:14 DEBUG }, 04:24:14 DEBUG "value": [ 04:24:14 DEBUG 1713583454, 04:24:14 DEBUG "651517952" 04:24:14 DEBUG ] 04:24:14 DEBUG } 04:24:14 DEBUG ] 04:24:14 DEBUG } 04:24:14 DEBUG } 04:24:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583454" 04:24:14 INFO [http_cmd]: http status code OK 04:24:14 DEBUG --- status code --- 04:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:14 DEBUG --- http response --- 04:24:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:14 DEBUG ----- output ----- 04:24:14 DEBUG { 04:24:14 DEBUG "status": "success", 04:24:14 DEBUG "data": { 04:24:14 DEBUG "resultType": "vector", 04:24:14 DEBUG "result": [ 04:24:14 DEBUG { 04:24:14 DEBUG "metric": { 04:24:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:24:14 DEBUG }, 04:24:14 DEBUG "value": [ 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG [...] 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:24:14 DEBUG }, 04:24:14 DEBUG "value": [ 04:24:14 DEBUG 1713583454, 04:24:14 DEBUG "0.02787037070828262" 04:24:14 DEBUG ] 04:24:14 DEBUG } 04:24:14 DEBUG ] 04:24:14 DEBUG } 04:24:14 DEBUG } 04:24:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583454" 04:24:14 INFO [http_cmd]: http status code OK 04:24:14 DEBUG --- status code --- 04:24:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:24:14 DEBUG --- http response --- 04:24:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:24:14 DEBUG ----- output ----- 04:24:14 DEBUG { 04:24:14 DEBUG "status": "success", 04:24:14 DEBUG "data": { 04:24:14 DEBUG "resultType": "vector", 04:24:14 DEBUG "result": [ 04:24:14 DEBUG { 04:24:14 DEBUG "metric": { 04:24:14 DEBUG "instance": "10.142.0.107:9100" 04:24:14 DEBUG }, 04:24:14 DEBUG "value": [ 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG [...] 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG 04:24:14 DEBUG "instance": "10.142.0.43:9100" 04:24:14 DEBUG }, 04:24:14 DEBUG "value": [ 04:24:14 DEBUG 1713583454, 04:24:14 DEBUG "0.052000000000007665" 04:24:14 DEBUG ] 04:24:14 DEBUG } 04:24:14 DEBUG ] 04:24:14 DEBUG } 04:24:14 DEBUG } 04:24:30 INFO 04:24:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:24:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:24:30 INFO [loop_until]: OK (rc = 0) 04:24:30 DEBUG --- stdout --- 04:24:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2559m 8610Mi am-767f6b67b5-qcqz2 2598m 8689Mi am-767f6b67b5-vbltz 3014m 8616Mi ds-cts-0 7770m 10596Mi ds-cts-1 7318m 10700Mi ds-cts-2 8367m 10599Mi ds-idrepo-0 770m 3694Mi ds-idrepo-1 770m 3625Mi ds-idrepo-2 749m 3522Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1m 526Mi 04:24:30 DEBUG --- stderr --- 04:24:30 DEBUG 04:24:34 WARNING File /tmp/lodemon_data-20240419_220641/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 04:24:34] "GET /monitoring/average?start_time=20240420_022348&stop_time=20240420_032233 HTTP/1.1" 200 - 04:24:35 INFO 04:24:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:24:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:24:35 INFO [loop_until]: OK (rc = 0) 04:24:35 DEBUG --- stdout --- 04:24:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6278Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 785m 2% 7362Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2520m 8% 11359Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7705m 25% 13756Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5188Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2653m 8% 11602Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2974Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3104m 10% 10861Mi 9% gke-xlou-cdm-ds-0f601751-j488 827m 2% 6823Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 7178m 24% 15614Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 850m 2% 7496Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8158m 27% 13245Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 76m 0% 4147Mi 7% 04:24:35 DEBUG --- stderr --- 04:24:35 DEBUG 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "143842" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:25:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "ds-cts-0" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "ds-cts-2" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "30" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "ds-idrepo-0" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "ds-idrepo-2" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0.03333333333333333" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "2669.6" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 ERROR [http_cmd]: ERROR 04:25:13 ERROR --- status code --- 04:25:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:25:13 ERROR --- http response --- 04:25:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:25:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "ds-idrepo-0" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "ds-idrepo-2" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0.03333333333333333" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "ds-idrepo-0" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "ds-idrepo-2" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0.03333333333333333" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "13108.259253272994" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0.082" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "instance": "10.142.0.107:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "instance": "10.142.0.43:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "17756642.366666667" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "instance": "10.142.0.107:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "instance": "10.142.0.43:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "17503560.833333332" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "instance": "10.142.0.107:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "instance": "10.142.0.43:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0.11180000000005919" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "instance": "10.142.0.107:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "instance": "10.142.0.43:9100" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "3036392537088" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "ds-cts-0" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "ds-cts-2" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0.009026216259900063" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583513" 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "ds-cts-0" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "ds-cts-2" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "0" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:13 INFO [http_cmd]: http status code OK 04:25:13 DEBUG --- status code --- 04:25:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:13 DEBUG --- http response --- 04:25:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:25:13 DEBUG ----- output ----- 04:25:13 DEBUG { 04:25:13 DEBUG "status": "success", 04:25:13 DEBUG "data": { 04:25:13 DEBUG "resultType": "vector", 04:25:13 DEBUG "result": [ 04:25:13 DEBUG { 04:25:13 DEBUG "metric": { 04:25:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG [...] 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG 04:25:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:25:13 DEBUG }, 04:25:13 DEBUG "value": [ 04:25:13 DEBUG 1713583513, 04:25:13 DEBUG "3839.7931145951193" 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG ] 04:25:13 DEBUG } 04:25:13 DEBUG } 04:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583514" 04:25:14 INFO [http_cmd]: http status code OK 04:25:14 DEBUG --- status code --- 04:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:14 DEBUG --- http response --- 04:25:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:25:14 DEBUG ----- output ----- 04:25:14 DEBUG { 04:25:14 DEBUG "status": "success", 04:25:14 DEBUG "data": { 04:25:14 DEBUG "resultType": "vector", 04:25:14 DEBUG "result": [ 04:25:14 DEBUG { 04:25:14 DEBUG "metric": { 04:25:14 DEBUG "pod": "ds-idrepo-0" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG [...] 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG "pod": "ds-cts-2" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 1713583514, 04:25:14 DEBUG "0" 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG } 04:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583514" 04:25:14 INFO [http_cmd]: http status code OK 04:25:14 DEBUG --- status code --- 04:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:14 DEBUG --- http response --- 04:25:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:14 DEBUG ----- output ----- 04:25:14 DEBUG { 04:25:14 DEBUG "status": "success", 04:25:14 DEBUG "data": { 04:25:14 DEBUG "resultType": "vector", 04:25:14 DEBUG "result": [ 04:25:14 DEBUG { 04:25:14 DEBUG "metric": { 04:25:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG [...] 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 1713583514, 04:25:14 DEBUG "650891264" 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG } 04:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583514" 04:25:14 INFO [http_cmd]: http status code OK 04:25:14 DEBUG --- status code --- 04:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:14 DEBUG --- http response --- 04:25:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:14 DEBUG ----- output ----- 04:25:14 DEBUG { 04:25:14 DEBUG "status": "success", 04:25:14 DEBUG "data": { 04:25:14 DEBUG "resultType": "vector", 04:25:14 DEBUG "result": [ 04:25:14 DEBUG { 04:25:14 DEBUG "metric": { 04:25:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG [...] 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 1713583514, 04:25:14 DEBUG "0.0287681795098726" 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG } 04:25:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583514" 04:25:14 INFO [http_cmd]: http status code OK 04:25:14 DEBUG --- status code --- 04:25:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:25:14 DEBUG --- http response --- 04:25:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:25:14 DEBUG ----- output ----- 04:25:14 DEBUG { 04:25:14 DEBUG "status": "success", 04:25:14 DEBUG "data": { 04:25:14 DEBUG "resultType": "vector", 04:25:14 DEBUG "result": [ 04:25:14 DEBUG { 04:25:14 DEBUG "metric": { 04:25:14 DEBUG "instance": "10.142.0.107:9100" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG [...] 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG 04:25:14 DEBUG "instance": "10.142.0.43:9100" 04:25:14 DEBUG }, 04:25:14 DEBUG "value": [ 04:25:14 DEBUG 1713583514, 04:25:14 DEBUG "0.059333333333345686" 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG ] 04:25:14 DEBUG } 04:25:14 DEBUG } 04:25:30 INFO 04:25:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:25:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:25:30 INFO [loop_until]: OK (rc = 0) 04:25:30 DEBUG --- stdout --- 04:25:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2562m 8616Mi am-767f6b67b5-qcqz2 2708m 8688Mi am-767f6b67b5-vbltz 3066m 8614Mi ds-cts-0 7713m 10580Mi ds-cts-1 6834m 10680Mi ds-cts-2 7805m 10600Mi ds-idrepo-0 1217m 3713Mi ds-idrepo-1 1229m 3639Mi ds-idrepo-2 1201m 3527Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1177Mi idm-b85f9d978-wnhxb 6m 1236Mi lodemon-df4b7b554-2gxfx 5m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 529m 782Mi 04:25:30 DEBUG --- stderr --- 04:25:30 DEBUG 04:25:35 INFO 04:25:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:25:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:25:35 INFO [loop_until]: OK (rc = 0) 04:25:35 DEBUG --- stdout --- 04:25:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 80m 0% 6279Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1205m 4% 7353Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2561m 8% 11356Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 98m 0% 3370Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 7680m 25% 13835Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5181Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2727m 9% 11588Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 123m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3147m 10% 10855Mi 9% gke-xlou-cdm-ds-0f601751-j488 1318m 4% 6792Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 6862m 22% 15614Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 1203m 4% 7486Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 7834m 26% 13243Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 832m 5% 4438Mi 7% 04:25:35 DEBUG --- stderr --- 04:25:35 DEBUG 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "ds-idrepo-0" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "ds-idrepo-2" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "6.7" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "789.8" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "0" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "ds-idrepo-0" 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "ds-idrepo-2" 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "8.9" 04:26:13 DEBUG "0" 04:26:13 DEBUG ] 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "11988" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "ds-cts-0" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "ds-cts-2" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "30" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:26:13 ERROR [http_cmd]: ERROR 04:26:13 ERROR --- status code --- 04:26:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:26:13 ERROR --- http response --- 04:26:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:26:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "0" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "0" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "ds-cts-2" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "0" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "ds-idrepo-0" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "ds-idrepo-2" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "62.9" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "ds-cts-2" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "85569.84478935698" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583573" 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "0.082" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "instance": "10.142.0.107:9100" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "instance": "10.142.0.43:9100" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "27704.3" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "instance": "10.142.0.107:9100" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "instance": "10.142.0.43:9100" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "26904.466666666667" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583573" 04:26:13 INFO [http_cmd]: http status code OK 04:26:13 DEBUG --- status code --- 04:26:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:13 DEBUG --- http response --- 04:26:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:13 DEBUG ----- output ----- 04:26:13 DEBUG { 04:26:13 DEBUG "status": "success", 04:26:13 DEBUG "data": { 04:26:13 DEBUG "resultType": "vector", 04:26:13 DEBUG "result": [ 04:26:13 DEBUG { 04:26:13 DEBUG "metric": { 04:26:13 DEBUG "instance": "10.142.0.107:9100" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG [...] 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG 04:26:13 DEBUG "instance": "10.142.0.43:9100" 04:26:13 DEBUG }, 04:26:13 DEBUG "value": [ 04:26:13 DEBUG 1713583573, 04:26:13 DEBUG "0.00176666666662868" 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG ] 04:26:13 DEBUG } 04:26:13 DEBUG } 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583574" 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "instance": "10.142.0.107:9100" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "instance": "10.142.0.43:9100" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "3036625413120" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583574" 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583574" 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "pod": "ds-cts-0" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583574" 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "pod": "ds-cts-2" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "0" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "pod": "ds-cts-2" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "4.297442939299267" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "pod": "ds-cts-0" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "pod": "ds-cts-2" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "0" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583574" 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "pod": "ds-cts-2" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "3604434.5898004435" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583574" 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "pod": "ds-idrepo-0" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "pod": "ds-cts-2" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "0" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583574" 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "695955456" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583574" 04:26:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583574" 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "0.8608871320310246" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:14 INFO [http_cmd]: http status code OK 04:26:14 DEBUG --- status code --- 04:26:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:26:14 DEBUG --- http response --- 04:26:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:26:14 DEBUG ----- output ----- 04:26:14 DEBUG { 04:26:14 DEBUG "status": "success", 04:26:14 DEBUG "data": { 04:26:14 DEBUG "resultType": "vector", 04:26:14 DEBUG "result": [ 04:26:14 DEBUG { 04:26:14 DEBUG "metric": { 04:26:14 DEBUG "instance": "10.142.0.107:9100" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG [...] 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG 04:26:14 DEBUG "instance": "10.142.0.43:9100" 04:26:14 DEBUG }, 04:26:14 DEBUG "value": [ 04:26:14 DEBUG 1713583574, 04:26:14 DEBUG "0.0016666666666651508" 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG ] 04:26:14 DEBUG } 04:26:14 DEBUG } 04:26:30 INFO 04:26:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:26:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:26:30 INFO [loop_until]: OK (rc = 0) 04:26:30 DEBUG --- stdout --- 04:26:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 8m 8598Mi am-767f6b67b5-qcqz2 7m 8678Mi am-767f6b67b5-vbltz 6m 8609Mi ds-cts-0 17m 10660Mi ds-cts-1 16m 10678Mi ds-cts-2 14m 10603Mi ds-idrepo-0 205m 3721Mi ds-idrepo-1 13m 3631Mi ds-idrepo-2 12m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1227m 853Mi 04:26:30 DEBUG --- stderr --- 04:26:30 DEBUG 04:26:35 INFO 04:26:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:26:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:26:35 INFO [loop_until]: OK (rc = 0) 04:26:35 DEBUG --- stdout --- 04:26:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6285Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 168m 0% 7342Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11350Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 72m 0% 13817Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5176Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 11575Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2959Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 10843Mi 9% gke-xlou-cdm-ds-0f601751-j488 68m 0% 6792Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 15594Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 97m 0% 7503Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 13223Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1132m 7% 4545Mi 7% 04:26:35 DEBUG --- stderr --- 04:26:35 DEBUG 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG { 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- http response --- 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG "data": { 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG { 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG "data": { 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "data": { 04:27:13 DEBUG { 04:27:13 DEBUG { 04:27:13 DEBUG "resultType": "vector", 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG { 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG }, 04:27:13 DEBUG "result": [ 04:27:13 DEBUG "data": { 04:27:13 DEBUG "data": { 04:27:13 DEBUG { 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG "data": { 04:27:13 DEBUG "value": [ 04:27:13 DEBUG { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "metric": { 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "result": [ 04:27:13 DEBUG "result": [ 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:27:13 DEBUG 04:27:13 DEBUG "pod": "ds-cts-0" 04:27:13 DEBUG { 04:27:13 DEBUG { 04:27:13 DEBUG }, 04:27:13 DEBUG { 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "metric": { 04:27:13 DEBUG [...] 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "pod": "ds-idrepo-0" 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG 04:27:13 DEBUG "0" 04:27:13 DEBUG "pod": "ds-cts-2" 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG ] 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG } 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "pod": "ds-idrepo-2" 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG ] 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG }, 04:27:13 DEBUG }, 04:27:13 DEBUG "0" 04:27:13 DEBUG }, 04:27:13 DEBUG } 04:27:13 DEBUG "29.933333333333334" 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "value": [ 04:27:13 DEBUG ] 04:27:13 DEBUG "value": [ 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG } 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG } 04:27:13 DEBUG "0" 04:27:13 DEBUG "1" 04:27:13 DEBUG ] 04:27:13 DEBUG "0" 04:27:13 DEBUG ] 04:27:13 DEBUG ] 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG ] 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "pod": "ds-idrepo-0" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "ds-idrepo-2" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "0" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583633" 04:27:13 ERROR [http_cmd]: ERROR 04:27:13 ERROR --- status code --- 04:27:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:27:13 ERROR --- http response --- 04:27:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:27:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG "data": { 04:27:13 DEBUG { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG "data": { 04:27:13 DEBUG { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "result": [ 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG { 04:27:13 DEBUG }, 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG 04:27:13 DEBUG }, 04:27:13 DEBUG 04:27:13 DEBUG "value": [ 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG }, 04:27:13 DEBUG "0" 04:27:13 DEBUG "value": [ 04:27:13 DEBUG ] 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG } 04:27:13 DEBUG "0" 04:27:13 DEBUG ] 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "0" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "pod": "ds-idrepo-0" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "ds-idrepo-2" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "0" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "0" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583633" 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "instance": "10.142.0.107:9100" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "instance": "10.142.0.43:9100" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "27861.4" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "0.0825" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583633" 04:27:13 INFO [http_cmd]: http status code OK 04:27:13 DEBUG --- status code --- 04:27:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:13 DEBUG --- http response --- 04:27:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:13 DEBUG ----- output ----- 04:27:13 DEBUG { 04:27:13 DEBUG "status": "success", 04:27:13 DEBUG "data": { 04:27:13 DEBUG "resultType": "vector", 04:27:13 DEBUG "result": [ 04:27:13 DEBUG { 04:27:13 DEBUG "metric": { 04:27:13 DEBUG "instance": "10.142.0.107:9100" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG [...] 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG 04:27:13 DEBUG "instance": "10.142.0.43:9100" 04:27:13 DEBUG }, 04:27:13 DEBUG "value": [ 04:27:13 DEBUG 1713583633, 04:27:13 DEBUG "26530.63333333333" 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG ] 04:27:13 DEBUG } 04:27:13 DEBUG } 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583634" 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583634" 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "instance": "10.142.0.107:9100" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "instance": "10.142.0.43:9100" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "3036634395648" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "instance": "10.142.0.107:9100" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "instance": "10.142.0.43:9100" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "0.0008333333333344702" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583634" 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583634" 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583634" 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583634" 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583634" 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "metric": { 04:27:14 DEBUG { 04:27:14 DEBUG "result": [ 04:27:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:27:14 DEBUG "metric": { 04:27:14 DEBUG { 04:27:14 DEBUG }, 04:27:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "value": [ 04:27:14 DEBUG }, 04:27:14 DEBUG "pod": "ds-cts-0" 04:27:14 DEBUG 04:27:14 DEBUG "value": [ 04:27:14 DEBUG }, 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG }, 04:27:14 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:27:14 DEBUG 04:27:14 DEBUG "value": [ 04:27:14 DEBUG }, 04:27:14 DEBUG "pod": "ds-cts-2" 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG }, 04:27:14 DEBUG "0" 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG ] 04:27:14 DEBUG "0" 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG "0" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG } 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "pod": "ds-cts-0" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "pod": "ds-cts-2" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "0" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "pod": "ds-idrepo-0" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "pod": "ds-cts-2" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "0" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583634" 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "994103296" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583634" 04:27:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583634" 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "1.236171696982532" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:14 INFO [http_cmd]: http status code OK 04:27:14 DEBUG --- status code --- 04:27:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:27:14 DEBUG --- http response --- 04:27:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:27:14 DEBUG ----- output ----- 04:27:14 DEBUG { 04:27:14 DEBUG "status": "success", 04:27:14 DEBUG "data": { 04:27:14 DEBUG "resultType": "vector", 04:27:14 DEBUG "result": [ 04:27:14 DEBUG { 04:27:14 DEBUG "metric": { 04:27:14 DEBUG "instance": "10.142.0.107:9100" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG [...] 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG 04:27:14 DEBUG "instance": "10.142.0.43:9100" 04:27:14 DEBUG }, 04:27:14 DEBUG "value": [ 04:27:14 DEBUG 1713583634, 04:27:14 DEBUG "0.0003333333333330302" 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG ] 04:27:14 DEBUG } 04:27:14 DEBUG } 04:27:30 INFO 04:27:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:27:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:27:30 INFO [loop_until]: OK (rc = 0) 04:27:30 DEBUG --- stdout --- 04:27:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 8m 8600Mi ds-cts-0 17m 10660Mi ds-cts-1 15m 10678Mi ds-cts-2 16m 10603Mi ds-idrepo-0 16m 3720Mi ds-idrepo-1 15m 3630Mi ds-idrepo-2 12m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 999m 1199Mi 04:27:30 DEBUG --- stderr --- 04:27:30 DEBUG 04:27:35 INFO 04:27:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:27:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:27:35 INFO [loop_until]: OK (rc = 0) 04:27:35 DEBUG --- stdout --- 04:27:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6299Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7369Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 11319Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 72m 0% 13806Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5195Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11563Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 2959Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 10834Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 6800Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15571Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 93m 0% 7503Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13219Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 4871Mi 8% 04:27:35 DEBUG --- stderr --- 04:27:35 DEBUG 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "ds-idrepo-0" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "ds-idrepo-2" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "1" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "ds-idrepo-0" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "ds-idrepo-2" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "ds-cts-0" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "ds-cts-2" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "30" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583693" 04:28:13 ERROR [http_cmd]: ERROR 04:28:13 ERROR --- status code --- 04:28:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:28:13 ERROR --- http response --- 04:28:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:28:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "ds-cts-2" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "ds-idrepo-0" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "ds-idrepo-2" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "ds-cts-2" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "306376477.9812104" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "instance": "10.142.0.107:9100" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "instance": "10.142.0.43:9100" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "28224.733333333334" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583693" 04:28:13 INFO [http_cmd]: http status code OK 04:28:13 DEBUG --- status code --- 04:28:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:13 DEBUG --- http response --- 04:28:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:28:13 DEBUG ----- output ----- 04:28:13 DEBUG { 04:28:13 DEBUG "status": "success", 04:28:13 DEBUG "data": { 04:28:13 DEBUG "resultType": "vector", 04:28:13 DEBUG "result": [ 04:28:13 DEBUG { 04:28:13 DEBUG "metric": { 04:28:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG [...] 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG 04:28:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:28:13 DEBUG }, 04:28:13 DEBUG "value": [ 04:28:13 DEBUG 1713583693, 04:28:13 DEBUG "0.084" 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG ] 04:28:13 DEBUG } 04:28:13 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "instance": "10.142.0.107:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "instance": "10.142.0.43:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "26538.73333333333" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583694" 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "instance": "10.142.0.107:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "instance": "10.142.0.43:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "3036635819008" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "instance": "10.142.0.107:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "instance": "10.142.0.43:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "0.0008666666666546992" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583694" 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "pod": "ds-cts-0" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "pod": "ds-cts-2" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "0" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "pod": "ds-cts-0" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "pod": "ds-cts-2" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "0" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583694" 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "pod": "ds-cts-2" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "10120.446265785731" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "pod": "ds-cts-2" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "0.00952978789405197" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "pod": "ds-idrepo-0" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "pod": "ds-cts-2" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "0" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "1351761920" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "instance": "10.142.0.107:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "instance": "10.142.0.43:9100" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "0.0003333333333330302" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583694" 04:28:14 INFO [http_cmd]: http status code OK 04:28:14 DEBUG --- status code --- 04:28:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:28:14 DEBUG --- http response --- 04:28:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:28:14 DEBUG ----- output ----- 04:28:14 DEBUG { 04:28:14 DEBUG "status": "success", 04:28:14 DEBUG "data": { 04:28:14 DEBUG "resultType": "vector", 04:28:14 DEBUG "result": [ 04:28:14 DEBUG { 04:28:14 DEBUG "metric": { 04:28:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG [...] 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG 04:28:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:28:14 DEBUG }, 04:28:14 DEBUG "value": [ 04:28:14 DEBUG 1713583694, 04:28:14 DEBUG "1.0264661857143846" 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG ] 04:28:14 DEBUG } 04:28:14 DEBUG } 04:28:30 INFO 04:28:30 INFO [loop_until]: kubectl --namespace=xlou top pods 04:28:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:28:30 INFO [loop_until]: OK (rc = 0) 04:28:30 DEBUG --- stdout --- 04:28:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8673Mi am-767f6b67b5-vbltz 6m 8600Mi ds-cts-0 15m 10659Mi ds-cts-1 17m 10680Mi ds-cts-2 16m 10603Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 13m 3630Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1014m 1487Mi 04:28:30 DEBUG --- stderr --- 04:28:30 DEBUG 04:28:35 INFO 04:28:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:28:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:28:35 INFO [loop_until]: OK (rc = 0) 04:28:35 DEBUG --- stdout --- 04:28:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6274Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 7365Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 11316Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 13776Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 11564Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 82m 0% 2961Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 10829Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 6815Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 15567Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 93m 0% 7491Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 74m 0% 13217Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1039m 6% 5167Mi 8% 04:28:35 DEBUG --- stderr --- 04:28:35 DEBUG 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG { 04:29:13 DEBUG { 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG "data": { 04:29:13 DEBUG "data": { 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG "result": [ 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG { 04:29:13 DEBUG { 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "data": { 04:29:13 DEBUG "data": { 04:29:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG }, 04:29:13 DEBUG }, 04:29:13 DEBUG "result": [ 04:29:13 DEBUG "result": [ 04:29:13 DEBUG "value": [ 04:29:13 DEBUG "value": [ 04:29:13 DEBUG { 04:29:13 DEBUG { 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "ds-idrepo-0" 04:29:13 DEBUG "pod": "ds-idrepo-0" 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG }, 04:29:13 DEBUG }, 04:29:13 DEBUG [...] 04:29:13 DEBUG [...] 04:29:13 DEBUG "value": [ 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:13 DEBUG [...] 04:29:13 DEBUG [...] 04:29:13 DEBUG }, 04:29:13 DEBUG }, 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "value": [ 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "0" 04:29:13 DEBUG "0" 04:29:13 DEBUG "pod": "ds-idrepo-2" 04:29:13 DEBUG "pod": "ds-idrepo-2" 04:29:13 DEBUG ] 04:29:13 DEBUG ] 04:29:13 DEBUG }, 04:29:13 DEBUG }, 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 DEBUG "value": [ 04:29:13 DEBUG "value": [ 04:29:13 DEBUG ] 04:29:13 DEBUG ] 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 DEBUG "0" 04:29:13 DEBUG "0" 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "ds-cts-0" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "ds-cts-2" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "30.03333333333333" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583753" 04:29:13 ERROR [http_cmd]: ERROR 04:29:13 ERROR --- status code --- 04:29:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:29:13 ERROR --- http response --- 04:29:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:29:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "ds-idrepo-0" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "ds-idrepo-2" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "pod": "ds-cts-2" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "0" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583753" 04:29:13 INFO [http_cmd]: http status code OK 04:29:13 DEBUG --- status code --- 04:29:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:13 DEBUG --- http response --- 04:29:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:13 DEBUG ----- output ----- 04:29:13 DEBUG { 04:29:13 DEBUG "status": "success", 04:29:13 DEBUG "data": { 04:29:13 DEBUG "resultType": "vector", 04:29:13 DEBUG "result": [ 04:29:13 DEBUG { 04:29:13 DEBUG "metric": { 04:29:13 DEBUG "instance": "10.142.0.107:9100" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG [...] 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG 04:29:13 DEBUG "instance": "10.142.0.43:9100" 04:29:13 DEBUG }, 04:29:13 DEBUG "value": [ 04:29:13 DEBUG 1713583753, 04:29:13 DEBUG "27812.4" 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG ] 04:29:13 DEBUG } 04:29:13 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0.084" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "instance": "10.142.0.107:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "instance": "10.142.0.43:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "28050.466666666667" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "instance": "10.142.0.107:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "instance": "10.142.0.43:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0.0007666666666532744" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "instance": "10.142.0.107:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "instance": "10.142.0.43:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "3036637436928" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583754" 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "pod": "ds-cts-0" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "pod": "ds-cts-2" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "pod": "ds-cts-2" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "pod": "ds-cts-0" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "pod": "ds-cts-2" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583754" 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "pod": "ds-idrepo-0" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "pod": "ds-cts-2" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "1653465088" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "instance": "10.142.0.107:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "instance": "10.142.0.43:9100" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "0.0006666666666660604" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583754" 04:29:14 INFO [http_cmd]: http status code OK 04:29:14 DEBUG --- status code --- 04:29:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:29:14 DEBUG --- http response --- 04:29:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:29:14 DEBUG ----- output ----- 04:29:14 DEBUG { 04:29:14 DEBUG "status": "success", 04:29:14 DEBUG "data": { 04:29:14 DEBUG "resultType": "vector", 04:29:14 DEBUG "result": [ 04:29:14 DEBUG { 04:29:14 DEBUG "metric": { 04:29:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG [...] 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG 04:29:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:29:14 DEBUG }, 04:29:14 DEBUG "value": [ 04:29:14 DEBUG 1713583754, 04:29:14 DEBUG "1.0289349920843576" 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG ] 04:29:14 DEBUG } 04:29:14 DEBUG } 04:29:31 INFO 04:29:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:29:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:29:31 INFO [loop_until]: OK (rc = 0) 04:29:31 DEBUG --- stdout --- 04:29:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 18m 10659Mi ds-cts-1 16m 10679Mi ds-cts-2 16m 10603Mi ds-idrepo-0 15m 3721Mi ds-idrepo-1 13m 3630Mi ds-idrepo-2 12m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1021m 1785Mi 04:29:31 DEBUG --- stderr --- 04:29:31 DEBUG 04:29:35 INFO 04:29:35 INFO [loop_until]: kubectl --namespace=xlou top node 04:29:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:29:35 INFO [loop_until]: OK (rc = 0) 04:29:35 DEBUG --- stdout --- 04:29:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6282Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 7362Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 11312Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 73m 0% 13778Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5182Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 11570Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 2950Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 10824Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6814Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15566Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 7493Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 13207Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 5477Mi 9% 04:29:35 DEBUG --- stderr --- 04:29:35 DEBUG 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "ds-idrepo-0" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "ds-idrepo-2" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "ds-idrepo-0" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "ds-idrepo-2" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "ds-cts-0" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "ds-cts-2" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "29.933333333333334" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583813" 04:30:13 ERROR [http_cmd]: ERROR 04:30:13 ERROR --- status code --- 04:30:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:30:13 ERROR --- http response --- 04:30:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:30:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG { 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "ds-idrepo-0" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "ds-idrepo-2" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583813" 04:30:13 INFO [http_cmd]: http status code OK 04:30:13 DEBUG --- status code --- 04:30:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:13 DEBUG --- http response --- 04:30:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:30:13 DEBUG ----- output ----- 04:30:13 DEBUG { 04:30:13 DEBUG "status": "success", 04:30:13 DEBUG "data": { 04:30:13 DEBUG "resultType": "vector", 04:30:13 DEBUG "result": [ 04:30:13 DEBUG { 04:30:13 DEBUG "metric": { 04:30:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG [...] 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG 04:30:13 DEBUG "pod": "ds-cts-2" 04:30:13 DEBUG }, 04:30:13 DEBUG "value": [ 04:30:13 DEBUG 1713583813, 04:30:13 DEBUG "0" 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG ] 04:30:13 DEBUG } 04:30:13 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "instance": "10.142.0.107:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "instance": "10.142.0.43:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "28167.8" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0.084" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "instance": "10.142.0.107:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "instance": "10.142.0.43:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "26672.6" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "instance": "10.142.0.107:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "instance": "10.142.0.43:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0.0008666666666537518" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583814" 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "instance": "10.142.0.107:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "instance": "10.142.0.43:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "3036638915584" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "pod": "ds-cts-0" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "pod": "ds-cts-2" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "pod": "ds-cts-0" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "pod": "ds-cts-2" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "pod": "ds-cts-2" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583814" 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "pod": "ds-idrepo-0" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "pod": "ds-cts-2" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "2053459968" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "instance": "10.142.0.107:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "instance": "10.142.0.43:9100" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "0.0006666666666660604" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583814" 04:30:14 INFO [http_cmd]: http status code OK 04:30:14 DEBUG --- status code --- 04:30:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:30:14 DEBUG --- http response --- 04:30:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:30:14 DEBUG ----- output ----- 04:30:14 DEBUG { 04:30:14 DEBUG "status": "success", 04:30:14 DEBUG "data": { 04:30:14 DEBUG "resultType": "vector", 04:30:14 DEBUG "result": [ 04:30:14 DEBUG { 04:30:14 DEBUG "metric": { 04:30:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG [...] 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG 04:30:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:30:14 DEBUG }, 04:30:14 DEBUG "value": [ 04:30:14 DEBUG 1713583814, 04:30:14 DEBUG "1.026033423925713" 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG ] 04:30:14 DEBUG } 04:30:14 DEBUG } 04:30:31 INFO 04:30:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:30:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:30:31 INFO [loop_until]: OK (rc = 0) 04:30:31 DEBUG --- stdout --- 04:30:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 8m 8600Mi ds-cts-0 16m 10659Mi ds-cts-1 16m 10680Mi ds-cts-2 16m 10603Mi ds-idrepo-0 14m 3720Mi ds-idrepo-1 11m 3630Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 788m 1844Mi 04:30:31 DEBUG --- stderr --- 04:30:31 DEBUG 04:30:36 INFO 04:30:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:30:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:30:36 INFO [loop_until]: OK (rc = 0) 04:30:36 DEBUG --- stdout --- 04:30:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 7373Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11298Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 13791Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5177Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 68m 0% 11572Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 10816Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6812Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15570Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 7486Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 70m 0% 13204Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 878m 5% 5486Mi 9% 04:30:36 DEBUG --- stderr --- 04:30:36 DEBUG 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG { 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG { 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG "data": { 04:31:13 DEBUG "data": { 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG { 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG "result": [ 04:31:13 DEBUG "result": [ 04:31:13 DEBUG { 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG { 04:31:13 DEBUG { 04:31:13 DEBUG { 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG "data": { 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "data": { 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG "data": { 04:31:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:13 DEBUG "pod": "ds-idrepo-0" 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG "result": [ 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG }, 04:31:13 DEBUG }, 04:31:13 DEBUG "result": [ 04:31:13 DEBUG { 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG "result": [ 04:31:13 DEBUG "value": [ 04:31:13 DEBUG "value": [ 04:31:13 DEBUG { 04:31:13 DEBUG "metric": { 04:31:13 DEBUG { 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:13 DEBUG "metric": { 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "pod": "ds-idrepo-0" 04:31:13 DEBUG }, 04:31:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:13 DEBUG { 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG }, 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG [...] 04:31:13 DEBUG [...] 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG "value": [ 04:31:13 DEBUG "data": { 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "result": [ 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG [...] 04:31:13 DEBUG 04:31:13 DEBUG { 04:31:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:13 DEBUG "pod": "ds-idrepo-2" 04:31:13 DEBUG [...] 04:31:13 DEBUG 04:31:13 DEBUG [...] 04:31:13 DEBUG "metric": { 04:31:13 DEBUG }, 04:31:13 DEBUG }, 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:13 DEBUG "value": [ 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG }, 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG 04:31:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:13 DEBUG 04:31:13 DEBUG "value": [ 04:31:13 DEBUG "0" 04:31:13 DEBUG "0" 04:31:13 DEBUG "pod": "ds-idrepo-2" 04:31:13 DEBUG }, 04:31:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:13 DEBUG 04:31:13 DEBUG ] 04:31:13 DEBUG ] 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG }, 04:31:13 DEBUG 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG ] 04:31:13 DEBUG ] 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "0" 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG [...] 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG "0" 04:31:13 DEBUG ] 04:31:13 DEBUG "0" 04:31:13 DEBUG 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG }, 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG "value": [ 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "0" 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583873" 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 ERROR [http_cmd]: ERROR 04:31:13 DEBUG --- status code --- 04:31:13 ERROR --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 ERROR --- http response --- 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG { 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG "data": { 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG "result": [ 04:31:13 DEBUG { 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "pod": "ds-cts-0" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG [...] 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "pod": "ds-cts-2" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "30" 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:31:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG { 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG "data": { 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG "result": [ 04:31:13 DEBUG { 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG [...] 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "0" 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG { 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG "data": { 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG "result": [ 04:31:13 DEBUG { 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG [...] 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "0" 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 INFO [http_cmd]: http status code OK 04:31:13 DEBUG --- status code --- 04:31:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:13 DEBUG --- http response --- 04:31:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:13 DEBUG ----- output ----- 04:31:13 DEBUG { 04:31:13 DEBUG "status": "success", 04:31:13 DEBUG "data": { 04:31:13 DEBUG "resultType": "vector", 04:31:13 DEBUG "result": [ 04:31:13 DEBUG { 04:31:13 DEBUG "metric": { 04:31:13 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG [...] 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG 04:31:13 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:31:13 DEBUG }, 04:31:13 DEBUG "value": [ 04:31:13 DEBUG 1713583873, 04:31:13 DEBUG "127462713.0466624" 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG ] 04:31:13 DEBUG } 04:31:13 DEBUG } 04:31:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583873" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "ds-idrepo-0" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "ds-idrepo-2" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583873, 04:31:14 DEBUG "0" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "8782990.668088524" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583874" 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "instance": "10.142.0.107:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "instance": "10.142.0.43:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "27319.63333333333" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0.084" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "instance": "10.142.0.107:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "instance": "10.142.0.43:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "26028.166666666664" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "instance": "10.142.0.107:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "instance": "10.142.0.43:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0.0007666666666674852" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583874" 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG { 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "ds-cts-0" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "ds-cts-2" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "instance": "10.142.0.107:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "instance": "10.142.0.43:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "3036640498688" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "ds-cts-0" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "ds-cts-2" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "1.4866572890499188" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "ds-idrepo-0" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "ds-cts-2" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "6453.731830785125" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "2028756992" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "instance": "10.142.0.107:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "instance": "10.142.0.43:9100" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0.0003333333333330302" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583874" 04:31:14 INFO [http_cmd]: http status code OK 04:31:14 DEBUG --- status code --- 04:31:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:31:14 DEBUG --- http response --- 04:31:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:31:14 DEBUG ----- output ----- 04:31:14 DEBUG { 04:31:14 DEBUG "status": "success", 04:31:14 DEBUG "data": { 04:31:14 DEBUG "resultType": "vector", 04:31:14 DEBUG "result": [ 04:31:14 DEBUG { 04:31:14 DEBUG "metric": { 04:31:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG [...] 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG 04:31:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:31:14 DEBUG }, 04:31:14 DEBUG "value": [ 04:31:14 DEBUG 1713583874, 04:31:14 DEBUG "0.8054361079085817" 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG ] 04:31:14 DEBUG } 04:31:14 DEBUG } 04:31:31 INFO 04:31:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:31:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:31:31 INFO [loop_until]: OK (rc = 0) 04:31:31 DEBUG --- stdout --- 04:31:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 6m 8600Mi ds-cts-0 19m 10659Mi ds-cts-1 17m 10680Mi ds-cts-2 15m 10603Mi ds-idrepo-0 14m 3722Mi ds-idrepo-1 34m 3630Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1064m 2079Mi 04:31:31 DEBUG --- stderr --- 04:31:31 DEBUG 04:31:36 INFO 04:31:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:31:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:31:36 INFO [loop_until]: OK (rc = 0) 04:31:36 DEBUG --- stdout --- 04:31:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6279Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 7367Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 11303Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 73m 0% 13784Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 5174Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 11556Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 2929Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 10813Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 6803Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15572Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 7483Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 70m 0% 13197Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1132m 7% 5879Mi 10% 04:31:36 DEBUG --- stderr --- 04:31:36 DEBUG 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583933" 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 ERROR [http_cmd]: ERROR 04:32:13 ERROR --- status code --- 04:32:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:32:13 ERROR --- http response --- 04:32:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:32:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 DEBUG --- status code --- 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG { 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG "data": { 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG "result": [ 04:32:13 DEBUG { 04:32:13 DEBUG "metric": { 04:32:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:13 DEBUG }, 04:32:13 DEBUG "value": [ 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG [...] 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:13 DEBUG }, 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583933" 04:32:13 DEBUG "value": [ 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG "0" 04:32:13 DEBUG ] 04:32:13 DEBUG } 04:32:13 DEBUG ] 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG { 04:32:13 DEBUG { 04:32:13 DEBUG { 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG { 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG "data": { 04:32:13 DEBUG "data": { 04:32:13 DEBUG "data": { 04:32:13 DEBUG "data": { 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG "result": [ 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG "result": [ 04:32:13 DEBUG "result": [ 04:32:13 DEBUG "result": [ 04:32:13 DEBUG { 04:32:13 DEBUG { 04:32:13 DEBUG { 04:32:13 DEBUG { 04:32:13 DEBUG { 04:32:13 DEBUG "metric": { 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG "metric": { 04:32:13 DEBUG "metric": { 04:32:13 DEBUG "metric": { 04:32:13 INFO [http_cmd]: http status code OK 04:32:13 DEBUG "pod": "ds-cts-0" 04:32:13 DEBUG "data": { 04:32:13 DEBUG "pod": "ds-idrepo-0" 04:32:13 DEBUG "pod": "ds-idrepo-0" 04:32:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:13 DEBUG --- status code --- 04:32:13 DEBUG }, 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG }, 04:32:13 DEBUG }, 04:32:13 DEBUG }, 04:32:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "result": [ 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "value": [ 04:32:13 DEBUG --- http response --- 04:32:13 DEBUG 04:32:13 DEBUG { 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:13 DEBUG "metric": { 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG ----- output ----- 04:32:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG [...] 04:32:13 DEBUG { 04:32:13 DEBUG }, 04:32:13 DEBUG [...] 04:32:13 DEBUG [...] 04:32:13 DEBUG [...] 04:32:13 DEBUG 04:32:13 DEBUG "status": "success", 04:32:13 DEBUG "value": [ 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG "data": { 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG "resultType": "vector", 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG 04:32:13 DEBUG "pod": "ds-cts-2" 04:32:13 DEBUG "result": [ 04:32:13 DEBUG 04:32:13 DEBUG "pod": "ds-idrepo-2" 04:32:13 DEBUG "pod": "ds-idrepo-2" 04:32:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:13 DEBUG }, 04:32:13 DEBUG { 04:32:13 DEBUG [...] 04:32:13 DEBUG }, 04:32:13 DEBUG }, 04:32:13 DEBUG }, 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "metric": { 04:32:13 DEBUG 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "value": [ 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:13 DEBUG 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG "30" 04:32:13 DEBUG }, 04:32:13 DEBUG 04:32:13 DEBUG "0" 04:32:13 DEBUG "0" 04:32:13 DEBUG "0" 04:32:13 DEBUG ] 04:32:13 DEBUG "value": [ 04:32:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:13 DEBUG ] 04:32:13 DEBUG ] 04:32:13 DEBUG ] 04:32:13 DEBUG } 04:32:13 DEBUG 04:32:13 DEBUG }, 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG ] 04:32:13 DEBUG 04:32:13 DEBUG "value": [ 04:32:13 DEBUG ] 04:32:13 DEBUG ] 04:32:13 DEBUG ] 04:32:13 DEBUG } 04:32:13 DEBUG 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG [...] 04:32:13 DEBUG "0" 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:13 DEBUG 04:32:13 DEBUG ] 04:32:13 DEBUG 04:32:13 DEBUG } 04:32:13 DEBUG 04:32:13 DEBUG ] 04:32:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:13 DEBUG } 04:32:13 DEBUG }, 04:32:13 DEBUG } 04:32:13 DEBUG "value": [ 04:32:13 DEBUG 1713583933, 04:32:13 DEBUG "0" 04:32:13 DEBUG ] 04:32:13 DEBUG } 04:32:13 DEBUG ] 04:32:13 DEBUG } 04:32:13 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "ds-idrepo-0" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-idrepo-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "394316188.0984467" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0.085" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "instance": "10.142.0.107:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "instance": "10.142.0.43:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "27635.899999999998" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "instance": "10.142.0.107:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "instance": "10.142.0.43:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "26539.166666666664" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "instance": "10.142.0.107:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "instance": "10.142.0.43:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0.0006999999999607098" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583934" 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "ds-cts-0" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "instance": "10.142.0.107:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "instance": "10.142.0.43:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "3036641891328" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "ds-cts-0" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0.015175367047573865" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "ds-idrepo-0" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "pod": "ds-cts-2" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "17721.540534789703" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "2275405824" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "instance": "10.142.0.107:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "instance": "10.142.0.43:9100" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "0" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583934" 04:32:14 INFO [http_cmd]: http status code OK 04:32:14 DEBUG --- status code --- 04:32:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:32:14 DEBUG --- http response --- 04:32:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:32:14 DEBUG ----- output ----- 04:32:14 DEBUG { 04:32:14 DEBUG "status": "success", 04:32:14 DEBUG "data": { 04:32:14 DEBUG "resultType": "vector", 04:32:14 DEBUG "result": [ 04:32:14 DEBUG { 04:32:14 DEBUG "metric": { 04:32:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG [...] 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG 04:32:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:32:14 DEBUG }, 04:32:14 DEBUG "value": [ 04:32:14 DEBUG 1713583934, 04:32:14 DEBUG "1.1639601345431563" 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG ] 04:32:14 DEBUG } 04:32:14 DEBUG } 04:32:31 INFO 04:32:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:32:31 INFO [loop_until]: OK (rc = 0) 04:32:31 DEBUG --- stdout --- 04:32:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8673Mi am-767f6b67b5-vbltz 8m 8600Mi ds-cts-0 17m 10660Mi ds-cts-1 16m 10679Mi ds-cts-2 15m 10603Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 13m 3630Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 6m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1001m 2636Mi 04:32:31 DEBUG --- stderr --- 04:32:31 DEBUG 04:32:36 INFO 04:32:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:32:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:32:36 INFO [loop_until]: OK (rc = 0) 04:32:36 DEBUG --- stdout --- 04:32:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6272Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 7361Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 11307Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3341Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 13772Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5200Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 11550Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 10825Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 6809Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 15567Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 7490Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13189Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1045m 6% 6424Mi 10% 04:32:36 DEBUG --- stderr --- 04:32:36 DEBUG 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "result": [ 04:33:13 DEBUG { 04:33:13 DEBUG "metric": { 04:33:13 DEBUG "pod": "ds-idrepo-0" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG "pod": "ds-idrepo-2" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "0" 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713583993" 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG { 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "result": [ 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "result": [ 04:33:13 DEBUG { 04:33:13 DEBUG "result": [ 04:33:13 DEBUG { 04:33:13 DEBUG "metric": { 04:33:13 DEBUG { 04:33:13 DEBUG "metric": { 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:13 DEBUG "metric": { 04:33:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG }, 04:33:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:13 DEBUG }, 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG "value": [ 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:13 DEBUG 04:33:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:13 DEBUG }, 04:33:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "0" 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "0" 04:33:13 DEBUG ] 04:33:13 DEBUG "0" 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 ERROR [http_cmd]: ERROR 04:33:13 ERROR --- status code --- 04:33:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:33:13 ERROR --- http response --- 04:33:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:33:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583993" 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "result": [ 04:33:13 DEBUG { 04:33:13 DEBUG "metric": { 04:33:13 DEBUG "pod": "ds-cts-0" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG "pod": "ds-cts-2" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "30" 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "result": [ 04:33:13 DEBUG { 04:33:13 DEBUG "metric": { 04:33:13 DEBUG "pod": "ds-idrepo-0" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG "pod": "ds-idrepo-2" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "0" 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:13 INFO [http_cmd]: http status code OK 04:33:13 DEBUG --- status code --- 04:33:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:13 DEBUG --- http response --- 04:33:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:13 DEBUG ----- output ----- 04:33:13 DEBUG { 04:33:13 DEBUG "status": "success", 04:33:13 DEBUG "data": { 04:33:13 DEBUG "resultType": "vector", 04:33:13 DEBUG "result": [ 04:33:13 DEBUG { 04:33:13 DEBUG "metric": { 04:33:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG [...] 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG 04:33:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:13 DEBUG }, 04:33:13 DEBUG "value": [ 04:33:13 DEBUG 1713583993, 04:33:13 DEBUG "0" 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG ] 04:33:13 DEBUG } 04:33:13 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "ds-idrepo-0" 04:33:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:14 DEBUG }, 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-idrepo-2" 04:33:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:14 DEBUG }, 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0" 04:33:14 DEBUG "0" 04:33:14 DEBUG ] 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG { 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG "metric": { 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:14 DEBUG { 04:33:14 DEBUG }, 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "value": [ 04:33:14 DEBUG "data": { 04:33:14 DEBUG 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG 04:33:14 DEBUG "result": [ 04:33:14 DEBUG 04:33:14 DEBUG { 04:33:14 DEBUG [...] 04:33:14 DEBUG "metric": { 04:33:14 DEBUG 04:33:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:33:14 DEBUG 04:33:14 DEBUG }, 04:33:14 DEBUG 04:33:14 DEBUG "value": [ 04:33:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:14 DEBUG 04:33:14 DEBUG }, 04:33:14 DEBUG 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG [...] 04:33:14 DEBUG "0" 04:33:14 DEBUG 04:33:14 DEBUG ] 04:33:14 DEBUG 04:33:14 DEBUG } 04:33:14 DEBUG 04:33:14 DEBUG ] 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG } 04:33:14 DEBUG }, 04:33:14 DEBUG } 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "392230035.627359" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0.085" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "instance": "10.142.0.107:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "instance": "10.142.0.43:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "27493.8" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "instance": "10.142.0.107:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "instance": "10.142.0.43:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "26521.2" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "instance": "10.142.0.107:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "instance": "10.142.0.43:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0.0009333333333415793" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583994" 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "ds-cts-0" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "ds-cts-0" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "instance": "10.142.0.107:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "instance": "10.142.0.43:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "3036643376128" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0.01522170094125791" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "ds-idrepo-0" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "pod": "ds-cts-2" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "17482.662527778757" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "2962087936" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "instance": "10.142.0.107:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "instance": "10.142.0.43:9100" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "0.0003333333333330302" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713583994" 04:33:14 INFO [http_cmd]: http status code OK 04:33:14 DEBUG --- status code --- 04:33:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:33:14 DEBUG --- http response --- 04:33:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:33:14 DEBUG ----- output ----- 04:33:14 DEBUG { 04:33:14 DEBUG "status": "success", 04:33:14 DEBUG "data": { 04:33:14 DEBUG "resultType": "vector", 04:33:14 DEBUG "result": [ 04:33:14 DEBUG { 04:33:14 DEBUG "metric": { 04:33:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG [...] 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG 04:33:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:33:14 DEBUG }, 04:33:14 DEBUG "value": [ 04:33:14 DEBUG 1713583994, 04:33:14 DEBUG "1.0366845806793306" 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG ] 04:33:14 DEBUG } 04:33:14 DEBUG } 04:33:31 INFO 04:33:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:33:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:33:31 INFO [loop_until]: OK (rc = 0) 04:33:31 DEBUG --- stdout --- 04:33:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 6m 8601Mi ds-cts-0 17m 10660Mi ds-cts-1 15m 10679Mi ds-cts-2 15m 10603Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 11m 3631Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-df4b7b554-2gxfx 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1005m 3185Mi 04:33:31 DEBUG --- stderr --- 04:33:31 DEBUG 04:33:36 INFO 04:33:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:33:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:33:36 INFO [loop_until]: OK (rc = 0) 04:33:36 DEBUG --- stdout --- 04:33:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6272Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 7359Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11300Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 13766Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5201Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11547Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 2946Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 10810Mi 9% gke-xlou-cdm-ds-0f601751-j488 69m 0% 6812Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 15571Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 7494Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 77m 0% 13186Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 6971Mi 11% 04:33:36 DEBUG --- stderr --- 04:33:36 DEBUG 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG { 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "data": { 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "result": [ 04:34:13 DEBUG { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG [...] 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG "0" 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG { 04:34:13 DEBUG { 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "data": { 04:34:13 DEBUG "data": { 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "result": [ 04:34:13 DEBUG "result": [ 04:34:13 DEBUG { 04:34:13 DEBUG { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "pod": "ds-idrepo-0" 04:34:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:13 DEBUG }, 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584053" 04:34:13 DEBUG 04:34:13 DEBUG [...] 04:34:13 DEBUG [...] 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG "pod": "ds-idrepo-2" 04:34:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:13 DEBUG }, 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG "0" 04:34:13 DEBUG "0" 04:34:13 DEBUG ] 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 DEBUG ] 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG { 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "data": { 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "result": [ 04:34:13 DEBUG { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG [...] 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG "0" 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG { 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "data": { 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "result": [ 04:34:13 DEBUG { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG [...] 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG "0" 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 ERROR [http_cmd]: ERROR 04:34:13 ERROR --- status code --- 04:34:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:34:13 ERROR --- http response --- 04:34:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:34:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:34:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584053" 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG { 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "data": { 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "result": [ 04:34:13 DEBUG { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "pod": "ds-cts-0" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG [...] 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG "pod": "ds-cts-2" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG "29.966666666666665" 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:13 INFO [http_cmd]: http status code OK 04:34:13 DEBUG --- status code --- 04:34:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:13 DEBUG --- http response --- 04:34:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:13 DEBUG ----- output ----- 04:34:13 DEBUG { 04:34:13 DEBUG "status": "success", 04:34:13 DEBUG "data": { 04:34:13 DEBUG "resultType": "vector", 04:34:13 DEBUG "result": [ 04:34:13 DEBUG { 04:34:13 DEBUG "metric": { 04:34:13 DEBUG "pod": "ds-idrepo-0" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG [...] 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG 04:34:13 DEBUG "pod": "ds-idrepo-2" 04:34:13 DEBUG }, 04:34:13 DEBUG "value": [ 04:34:13 DEBUG 1713584053, 04:34:13 DEBUG "0" 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG ] 04:34:13 DEBUG } 04:34:13 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "ds-idrepo-0" 04:34:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:14 DEBUG }, 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-idrepo-2" 04:34:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:14 DEBUG }, 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0" 04:34:14 DEBUG "0" 04:34:14 DEBUG ] 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "6325.665197387471" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0.085" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "instance": "10.142.0.107:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "instance": "10.142.0.43:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "27971.7" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "instance": "10.142.0.107:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "instance": "10.142.0.43:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "27999.86666666667" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584054" 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "instance": "10.142.0.107:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "instance": "10.142.0.43:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0.0005333333334145133" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "instance": "10.142.0.107:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "instance": "10.142.0.43:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "3036645329920" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "15550.593610244201" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "ds-cts-0" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "ds-idrepo-0" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "ds-cts-0" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "pod": "ds-cts-2" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0.014809336896105656" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "3559165952" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "1.0264391988708848" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584054" 04:34:14 INFO [http_cmd]: http status code OK 04:34:14 DEBUG --- status code --- 04:34:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:34:14 DEBUG --- http response --- 04:34:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:34:14 DEBUG ----- output ----- 04:34:14 DEBUG { 04:34:14 DEBUG "status": "success", 04:34:14 DEBUG "data": { 04:34:14 DEBUG "resultType": "vector", 04:34:14 DEBUG "result": [ 04:34:14 DEBUG { 04:34:14 DEBUG "metric": { 04:34:14 DEBUG "instance": "10.142.0.107:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG [...] 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG 04:34:14 DEBUG "instance": "10.142.0.43:9100" 04:34:14 DEBUG }, 04:34:14 DEBUG "value": [ 04:34:14 DEBUG 1713584054, 04:34:14 DEBUG "0.0006666666666660604" 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG ] 04:34:14 DEBUG } 04:34:14 DEBUG } 04:34:31 INFO 04:34:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:34:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:34:31 INFO [loop_until]: OK (rc = 0) 04:34:31 DEBUG --- stdout --- 04:34:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8673Mi am-767f6b67b5-vbltz 7m 8601Mi ds-cts-0 16m 10659Mi ds-cts-1 15m 10679Mi ds-cts-2 16m 10603Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 13m 3630Mi ds-idrepo-2 15m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1003m 3672Mi 04:34:31 DEBUG --- stderr --- 04:34:31 DEBUG 04:34:36 INFO 04:34:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:34:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:34:36 INFO [loop_until]: OK (rc = 0) 04:34:36 DEBUG --- stdout --- 04:34:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6269Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 7362Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 11306Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 73m 0% 13750Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 11550Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 75m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 10801Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 6809Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15567Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 93m 0% 7490Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 13189Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 7516Mi 12% 04:34:36 DEBUG --- stderr --- 04:34:36 DEBUG 04:35:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584113" 04:35:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584113" 04:35:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584113" 04:35:13 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584113" 04:35:13 INFO [http_cmd]: http status code OK 04:35:13 INFO [http_cmd]: http status code OK 04:35:13 DEBUG --- status code --- 04:35:13 ERROR [http_cmd]: ERROR 04:35:13 DEBUG --- status code --- 04:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:13 ERROR --- status code --- 04:35:13 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:13 DEBUG --- http response --- 04:35:13 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:35:13 DEBUG --- http response --- 04:35:13 ERROR --- http response --- 04:35:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:13 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:13 DEBUG ----- output ----- 04:35:13 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:35:13 DEBUG ----- output ----- 04:35:13 DEBUG { 04:35:13 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:35:13 DEBUG { 04:35:13 DEBUG "status": "success", 04:35:13 DEBUG "status": "success", 04:35:13 DEBUG "data": { 04:35:13 DEBUG "data": { 04:35:13 DEBUG "resultType": "vector", 04:35:13 DEBUG "resultType": "vector", 04:35:13 DEBUG "result": [ 04:35:13 DEBUG "result": [ 04:35:13 DEBUG { 04:35:13 DEBUG { 04:35:13 DEBUG "metric": { 04:35:13 DEBUG "metric": { 04:35:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:13 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:13 DEBUG }, 04:35:13 DEBUG }, 04:35:13 DEBUG "value": [ 04:35:13 DEBUG "value": [ 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG [...] 04:35:13 DEBUG [...] 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG 04:35:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:13 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:13 DEBUG }, 04:35:13 DEBUG }, 04:35:13 DEBUG "value": [ 04:35:13 DEBUG "value": [ 04:35:13 DEBUG 1713584113, 04:35:13 DEBUG 1713584113, 04:35:13 DEBUG "0" 04:35:13 DEBUG "0" 04:35:13 DEBUG ] 04:35:13 DEBUG ] 04:35:13 DEBUG } 04:35:13 DEBUG } 04:35:13 DEBUG ] 04:35:13 DEBUG ] 04:35:13 DEBUG } 04:35:13 DEBUG } 04:35:13 DEBUG } 04:35:13 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584113, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-idrepo-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-idrepo-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:35:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-cts-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-cts-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "30" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-idrepo-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-idrepo-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-idrepo-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-idrepo-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "2493.336653461611" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0.085" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "instance": "10.142.0.107:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "instance": "10.142.0.43:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "27709.333333333332" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "instance": "10.142.0.107:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "instance": "10.142.0.43:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "26500.13333333333" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "instance": "10.142.0.107:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "instance": "10.142.0.43:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0.0007999999999877143" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "instance": "10.142.0.107:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "instance": "10.142.0.43:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "3036646695936" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-cts-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-cts-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-idrepo-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-cts-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "ds-cts-0" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "ds-cts-2" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0.0041968252499525556" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "4120526848" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "1.040126005917199" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584114" 04:35:14 INFO [http_cmd]: http status code OK 04:35:14 DEBUG --- status code --- 04:35:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:35:14 DEBUG --- http response --- 04:35:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:35:14 DEBUG ----- output ----- 04:35:14 DEBUG { 04:35:14 DEBUG "status": "success", 04:35:14 DEBUG "data": { 04:35:14 DEBUG "resultType": "vector", 04:35:14 DEBUG "result": [ 04:35:14 DEBUG { 04:35:14 DEBUG "metric": { 04:35:14 DEBUG "instance": "10.142.0.107:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG [...] 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG 04:35:14 DEBUG "instance": "10.142.0.43:9100" 04:35:14 DEBUG }, 04:35:14 DEBUG "value": [ 04:35:14 DEBUG 1713584114, 04:35:14 DEBUG "0.0006666666666660604" 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG ] 04:35:14 DEBUG } 04:35:14 DEBUG } 04:35:31 INFO 04:35:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:35:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:35:31 INFO [loop_until]: OK (rc = 0) 04:35:31 DEBUG --- stdout --- 04:35:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 8m 8600Mi ds-cts-0 17m 10661Mi ds-cts-1 17m 10679Mi ds-cts-2 16m 10603Mi ds-idrepo-0 14m 3720Mi ds-idrepo-1 13m 3631Mi ds-idrepo-2 12m 3527Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 5m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1020m 4276Mi 04:35:31 DEBUG --- stderr --- 04:35:31 DEBUG 04:35:36 INFO 04:35:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:35:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:35:36 INFO [loop_until]: OK (rc = 0) 04:35:36 DEBUG --- stdout --- 04:35:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6276Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 75m 0% 7367Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11310Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3346Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 72m 0% 13776Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 11534Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 2941Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 10796Mi 9% gke-xlou-cdm-ds-0f601751-j488 69m 0% 6817Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 15565Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 94m 0% 7502Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13190Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1074m 6% 8068Mi 13% 04:35:36 DEBUG --- stderr --- 04:35:36 DEBUG 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG { 04:36:14 ERROR [http_cmd]: ERROR 04:36:14 DEBUG "metric": { 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG "metric": { 04:36:14 ERROR --- status code --- 04:36:14 DEBUG "pod": "ds-idrepo-0" 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG "pod": "ds-idrepo-0" 04:36:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:36:14 DEBUG }, 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG }, 04:36:14 ERROR --- http response --- 04:36:14 DEBUG "value": [ 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:36:14 DEBUG 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG 04:36:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:36:14 DEBUG 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG { 04:36:14 DEBUG [...] 04:36:14 DEBUG [...] 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "data": { 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "result": [ 04:36:14 DEBUG "pod": "ds-idrepo-2" 04:36:14 DEBUG "pod": "ds-idrepo-2" 04:36:14 DEBUG { 04:36:14 DEBUG }, 04:36:14 DEBUG }, 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "value": [ 04:36:14 DEBUG "value": [ 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG 1713584174, 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG }, 04:36:14 DEBUG "0" 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG "0" 04:36:14 DEBUG "value": [ 04:36:14 DEBUG ] 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG ] 04:36:14 DEBUG 04:36:14 DEBUG } 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG } 04:36:14 DEBUG 04:36:14 DEBUG ] 04:36:14 DEBUG ] 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG [...] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 DEBUG { 04:36:14 DEBUG 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG 04:36:14 DEBUG "data": { 04:36:14 DEBUG 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG "result": [ 04:36:14 DEBUG }, 04:36:14 DEBUG { 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG "value": [ 04:36:14 DEBUG "metric": { 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG "0" 04:36:14 DEBUG }, 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG ] 04:36:14 DEBUG "value": [ 04:36:14 DEBUG } 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG 04:36:14 DEBUG ] 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG 04:36:14 DEBUG } 04:36:14 DEBUG { 04:36:14 DEBUG 04:36:14 DEBUG } 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG "data": { 04:36:14 DEBUG 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG 04:36:14 DEBUG "result": [ 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG { 04:36:14 DEBUG }, 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "value": [ 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG }, 04:36:14 DEBUG "0" 04:36:14 DEBUG "value": [ 04:36:14 DEBUG ] 04:36:14 DEBUG 04:36:14 DEBUG } 04:36:14 DEBUG 04:36:14 DEBUG ] 04:36:14 DEBUG 04:36:14 DEBUG } 04:36:14 DEBUG [...] 04:36:14 DEBUG } 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "ds-cts-0" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "ds-cts-2" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "18.366666666666667" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "ds-idrepo-0" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "ds-idrepo-2" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "19212196.003666036" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0.085" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "instance": "10.142.0.107:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "instance": "10.142.0.43:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "27918.5" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "instance": "10.142.0.107:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "instance": "10.142.0.43:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "26402.13333333333" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "instance": "10.142.0.107:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "instance": "10.142.0.43:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0.0007333333333273609" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "instance": "10.142.0.107:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "instance": "10.142.0.43:9100" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "3036648076288" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "ds-cts-0" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "ds-cts-2" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "ds-idrepo-0" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "ds-cts-2" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "ds-cts-0" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "ds-cts-2" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "0" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "2.0073857789399234" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584174" 04:36:14 INFO [http_cmd]: http status code OK 04:36:14 DEBUG --- status code --- 04:36:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:14 DEBUG --- http response --- 04:36:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:14 DEBUG ----- output ----- 04:36:14 DEBUG { 04:36:14 DEBUG "status": "success", 04:36:14 DEBUG "data": { 04:36:14 DEBUG "resultType": "vector", 04:36:14 DEBUG "result": [ 04:36:14 DEBUG { 04:36:14 DEBUG "metric": { 04:36:14 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG [...] 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG 04:36:14 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:36:14 DEBUG }, 04:36:14 DEBUG "value": [ 04:36:14 DEBUG 1713584174, 04:36:14 DEBUG "4579872768" 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG ] 04:36:14 DEBUG } 04:36:14 DEBUG } 04:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584175" 04:36:15 INFO [http_cmd]: http status code OK 04:36:15 DEBUG --- status code --- 04:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:15 DEBUG --- http response --- 04:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:15 DEBUG ----- output ----- 04:36:15 DEBUG { 04:36:15 DEBUG "status": "success", 04:36:15 DEBUG "data": { 04:36:15 DEBUG "resultType": "vector", 04:36:15 DEBUG "result": [ 04:36:15 DEBUG { 04:36:15 DEBUG "metric": { 04:36:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:36:15 DEBUG }, 04:36:15 DEBUG "value": [ 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG [...] 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:36:15 DEBUG }, 04:36:15 DEBUG "value": [ 04:36:15 DEBUG 1713584175, 04:36:15 DEBUG "1.0344388663715385" 04:36:15 DEBUG ] 04:36:15 DEBUG } 04:36:15 DEBUG ] 04:36:15 DEBUG } 04:36:15 DEBUG } 04:36:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584175" 04:36:15 INFO [http_cmd]: http status code OK 04:36:15 DEBUG --- status code --- 04:36:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:36:15 DEBUG --- http response --- 04:36:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:36:15 DEBUG ----- output ----- 04:36:15 DEBUG { 04:36:15 DEBUG "status": "success", 04:36:15 DEBUG "data": { 04:36:15 DEBUG "resultType": "vector", 04:36:15 DEBUG "result": [ 04:36:15 DEBUG { 04:36:15 DEBUG "metric": { 04:36:15 DEBUG "instance": "10.142.0.107:9100" 04:36:15 DEBUG }, 04:36:15 DEBUG "value": [ 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG [...] 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG 04:36:15 DEBUG "instance": "10.142.0.43:9100" 04:36:15 DEBUG }, 04:36:15 DEBUG "value": [ 04:36:15 DEBUG 1713584175, 04:36:15 DEBUG "0.0003333333333330302" 04:36:15 DEBUG ] 04:36:15 DEBUG } 04:36:15 DEBUG ] 04:36:15 DEBUG } 04:36:15 DEBUG } 04:36:31 INFO 04:36:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:36:31 INFO [loop_until]: OK (rc = 0) 04:36:31 DEBUG --- stdout --- 04:36:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 21m 10570Mi ds-cts-1 20m 10677Mi ds-cts-2 15m 10600Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 11m 3630Mi ds-idrepo-2 11m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 368m 4366Mi 04:36:31 DEBUG --- stderr --- 04:36:31 DEBUG 04:36:36 INFO 04:36:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:36:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:36:36 INFO [loop_until]: OK (rc = 0) 04:36:36 DEBUG --- stdout --- 04:36:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6281Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 7360Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 11308Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 74m 0% 13668Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11531Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2934Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 10801Mi 9% gke-xlou-cdm-ds-0f601751-j488 69m 0% 6810Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 15560Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 7501Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 13185Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 454m 2% 8324Mi 14% 04:36:36 DEBUG --- stderr --- 04:36:36 DEBUG 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 ERROR [http_cmd]: ERROR 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 ERROR --- status code --- 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 DEBUG --- status code --- 04:37:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 ERROR --- http response --- 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG "data": { 04:37:14 DEBUG "data": { 04:37:14 DEBUG { 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG "result": [ 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG "result": [ 04:37:14 DEBUG "data": { 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "result": [ 04:37:14 DEBUG "data": { 04:37:14 DEBUG "pod": "ds-idrepo-0" 04:37:14 DEBUG "data": { 04:37:14 DEBUG "pod": "ds-idrepo-0" 04:37:14 DEBUG { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG }, 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG }, 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "result": [ 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "result": [ 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG { 04:37:14 DEBUG 04:37:14 DEBUG { 04:37:14 DEBUG 04:37:14 DEBUG }, 04:37:14 DEBUG "metric": { 04:37:14 DEBUG 04:37:14 DEBUG "metric": { 04:37:14 DEBUG 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG }, 04:37:14 DEBUG [...] 04:37:14 DEBUG }, 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-idrepo-2" 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-idrepo-2" 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG }, 04:37:14 DEBUG [...] 04:37:14 DEBUG }, 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG }, 04:37:14 DEBUG 04:37:14 DEBUG "0" 04:37:14 DEBUG 04:37:14 DEBUG "0" 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG ] 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG ] 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG }, 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG }, 04:37:14 DEBUG } 04:37:14 DEBUG "0" 04:37:14 DEBUG "value": [ 04:37:14 DEBUG ] 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG "value": [ 04:37:14 DEBUG ] 04:37:14 DEBUG ] 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG } 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG "0" 04:37:14 DEBUG } 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG "0" 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG ] 04:37:14 DEBUG ] 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG ] 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG { 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG "data": { 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "data": { 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG }, 04:37:14 DEBUG "result": [ 04:37:14 DEBUG "value": [ 04:37:14 DEBUG { 04:37:14 DEBUG 04:37:14 DEBUG "metric": { 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-cts-0" 04:37:14 DEBUG 04:37:14 DEBUG }, 04:37:14 DEBUG [...] 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG [...] 04:37:14 DEBUG }, 04:37:14 DEBUG 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG 04:37:14 DEBUG "0" 04:37:14 DEBUG "pod": "ds-cts-2" 04:37:14 DEBUG ] 04:37:14 DEBUG }, 04:37:14 DEBUG } 04:37:14 DEBUG "value": [ 04:37:14 DEBUG ] 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG } 04:37:14 DEBUG "0" 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "ds-idrepo-0" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-idrepo-2" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "1866969.0404294953" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0.085" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "instance": "10.142.0.107:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "instance": "10.142.0.43:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "27599.266666666666" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "instance": "10.142.0.107:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "instance": "10.142.0.43:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "26579.300000000003" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "instance": "10.142.0.107:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "instance": "10.142.0.43:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0.00033333333336050446" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "ds-idrepo-0" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-cts-2" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584234" 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "ds-cts-0" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-cts-2" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "ds-cts-0" 04:37:14 DEBUG "instance": "10.142.0.107:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "ds-cts-2" 04:37:14 DEBUG "instance": "10.142.0.43:9100" 04:37:14 DEBUG }, 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG "3036651070464" 04:37:14 DEBUG ] 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584234" 04:37:14 INFO [http_cmd]: http status code OK 04:37:14 DEBUG --- status code --- 04:37:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:14 DEBUG --- http response --- 04:37:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:14 DEBUG ----- output ----- 04:37:14 DEBUG { 04:37:14 DEBUG "status": "success", 04:37:14 DEBUG "data": { 04:37:14 DEBUG "resultType": "vector", 04:37:14 DEBUG "result": [ 04:37:14 DEBUG { 04:37:14 DEBUG "metric": { 04:37:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG [...] 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG 04:37:14 DEBUG "pod": "lodemon-df4b7b554-2gxfx" 04:37:14 DEBUG }, 04:37:14 DEBUG "value": [ 04:37:14 DEBUG 1713584234, 04:37:14 DEBUG "0.0048466671578832185" 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG ] 04:37:14 DEBUG } 04:37:14 DEBUG } 04:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584235" 04:37:15 INFO [http_cmd]: http status code OK 04:37:15 DEBUG --- status code --- 04:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:15 DEBUG --- http response --- 04:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:15 DEBUG ----- output ----- 04:37:15 DEBUG { 04:37:15 DEBUG "status": "success", 04:37:15 DEBUG "data": { 04:37:15 DEBUG "resultType": "vector", 04:37:15 DEBUG "result": [ 04:37:15 DEBUG { 04:37:15 DEBUG "metric": { 04:37:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:37:15 DEBUG }, 04:37:15 DEBUG "value": [ 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG [...] 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:37:15 DEBUG }, 04:37:15 DEBUG "value": [ 04:37:15 DEBUG 1713584235, 04:37:15 DEBUG "4891353088" 04:37:15 DEBUG ] 04:37:15 DEBUG } 04:37:15 DEBUG ] 04:37:15 DEBUG } 04:37:15 DEBUG } 04:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584235" 04:37:15 INFO [http_cmd]: http status code OK 04:37:15 DEBUG --- status code --- 04:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:15 DEBUG --- http response --- 04:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:15 DEBUG ----- output ----- 04:37:15 DEBUG { 04:37:15 DEBUG "status": "success", 04:37:15 DEBUG "data": { 04:37:15 DEBUG "resultType": "vector", 04:37:15 DEBUG "result": [ 04:37:15 DEBUG { 04:37:15 DEBUG "metric": { 04:37:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:37:15 DEBUG }, 04:37:15 DEBUG "value": [ 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG [...] 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:37:15 DEBUG }, 04:37:15 DEBUG "value": [ 04:37:15 DEBUG 1713584235, 04:37:15 DEBUG "0.5879871263797736" 04:37:15 DEBUG ] 04:37:15 DEBUG } 04:37:15 DEBUG ] 04:37:15 DEBUG } 04:37:15 DEBUG } 04:37:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584235" 04:37:15 INFO [http_cmd]: http status code OK 04:37:15 DEBUG --- status code --- 04:37:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:37:15 DEBUG --- http response --- 04:37:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:37:15 DEBUG ----- output ----- 04:37:15 DEBUG { 04:37:15 DEBUG "status": "success", 04:37:15 DEBUG "data": { 04:37:15 DEBUG "resultType": "vector", 04:37:15 DEBUG "result": [ 04:37:15 DEBUG { 04:37:15 DEBUG "metric": { 04:37:15 DEBUG "instance": "10.142.0.107:9100" 04:37:15 DEBUG }, 04:37:15 DEBUG "value": [ 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG [...] 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG 04:37:15 DEBUG "instance": "10.142.0.43:9100" 04:37:15 DEBUG }, 04:37:15 DEBUG "value": [ 04:37:15 DEBUG 1713584235, 04:37:15 DEBUG "0" 04:37:15 DEBUG ] 04:37:15 DEBUG } 04:37:15 DEBUG ] 04:37:15 DEBUG } 04:37:15 DEBUG } 04:37:31 INFO 04:37:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:37:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:37:31 INFO [loop_until]: OK (rc = 0) 04:37:31 DEBUG --- stdout --- 04:37:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 14m 10570Mi ds-cts-1 13m 10676Mi ds-cts-2 12m 10600Mi ds-idrepo-0 14m 3722Mi ds-idrepo-1 13m 3630Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1047m 4783Mi 04:37:31 DEBUG --- stderr --- 04:37:31 DEBUG 04:37:36 INFO 04:37:36 INFO [loop_until]: kubectl --namespace=xlou top node 04:37:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:37:36 INFO [loop_until]: OK (rc = 0) 04:37:36 DEBUG --- stdout --- 04:37:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6261Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 7370Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 11300Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3346Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 13685Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5216Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 69m 0% 11524Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 10806Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 6815Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 15557Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 7494Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 13184Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 8659Mi 14% 04:37:36 DEBUG --- stderr --- 04:37:36 DEBUG 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 ERROR [http_cmd]: ERROR 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 ERROR --- status code --- 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 DEBUG --- status code --- 04:38:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 ERROR --- http response --- 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:38:14 DEBUG --- status code --- 04:38:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "ds-idrepo-0" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG "pod": "ds-idrepo-2" 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG }, 04:38:14 DEBUG { 04:38:14 DEBUG "value": [ 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "data": { 04:38:14 DEBUG "0" 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG ] 04:38:14 DEBUG "result": [ 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG { 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG ] 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG "metric": { 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG } 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG } 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG }, 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG "value": [ 04:38:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:38:14 DEBUG 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG 04:38:14 DEBUG ----- output ----- 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG 04:38:14 DEBUG { 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG [...] 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG 04:38:14 DEBUG "data": { 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG 04:38:14 DEBUG "result": [ 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 DEBUG { 04:38:14 DEBUG { 04:38:14 DEBUG }, 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "value": [ 04:38:14 DEBUG "pod": "ds-idrepo-0" 04:38:14 DEBUG "data": { 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG }, 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "0" 04:38:14 DEBUG "value": [ 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG "result": [ 04:38:14 DEBUG ] 04:38:14 DEBUG 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG { 04:38:14 DEBUG } 04:38:14 DEBUG 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG "metric": { 04:38:14 DEBUG ] 04:38:14 DEBUG 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 DEBUG } 04:38:14 DEBUG [...] 04:38:14 DEBUG }, 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG } 04:38:14 DEBUG 04:38:14 DEBUG "value": [ 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG { 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "pod": "ds-idrepo-2" 04:38:14 DEBUG 04:38:14 DEBUG "data": { 04:38:14 DEBUG }, 04:38:14 DEBUG [...] 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG "result": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG 04:38:14 DEBUG { 04:38:14 DEBUG "0" 04:38:14 DEBUG 04:38:14 DEBUG "metric": { 04:38:14 DEBUG ] 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG } 04:38:14 DEBUG }, 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG }, 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG ] 04:38:14 DEBUG "value": [ 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG "value": [ 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG } 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG } 04:38:14 DEBUG "0" 04:38:14 DEBUG 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG ] 04:38:14 DEBUG 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG } 04:38:14 DEBUG [...] 04:38:14 DEBUG { 04:38:14 DEBUG { 04:38:14 DEBUG ] 04:38:14 DEBUG 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG } 04:38:14 DEBUG 04:38:14 DEBUG "data": { 04:38:14 DEBUG "data": { 04:38:14 DEBUG } 04:38:14 DEBUG 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 DEBUG "result": [ 04:38:14 DEBUG "result": [ 04:38:14 DEBUG }, 04:38:14 DEBUG { 04:38:14 DEBUG { 04:38:14 DEBUG "value": [ 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "pod": "ds-cts-0" 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 DEBUG "0" 04:38:14 DEBUG }, 04:38:14 DEBUG }, 04:38:14 DEBUG ] 04:38:14 DEBUG "value": [ 04:38:14 DEBUG "value": [ 04:38:14 DEBUG } 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG ] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG } 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG } 04:38:14 DEBUG [...] 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 DEBUG }, 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "ds-idrepo-0" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-idrepo-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "366733541.9127107" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0.085" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "instance": "10.142.0.107:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "instance": "10.142.0.43:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "27659.2" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "instance": "10.142.0.107:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "instance": "10.142.0.43:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "26399.433333333334" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "instance": "10.142.0.107:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "instance": "10.142.0.43:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0.0004333333333467711" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "ds-cts-0" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "ds-cts-0" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "ds-idrepo-0" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "instance": "10.142.0.107:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "instance": "10.142.0.43:9100" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "3036651486208" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "675.6177217695379" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584294" 04:38:14 INFO [http_cmd]: http status code OK 04:38:14 DEBUG --- status code --- 04:38:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:14 DEBUG --- http response --- 04:38:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:38:14 DEBUG ----- output ----- 04:38:14 DEBUG { 04:38:14 DEBUG "status": "success", 04:38:14 DEBUG "data": { 04:38:14 DEBUG "resultType": "vector", 04:38:14 DEBUG "result": [ 04:38:14 DEBUG { 04:38:14 DEBUG "metric": { 04:38:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG [...] 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG 04:38:14 DEBUG "pod": "ds-cts-2" 04:38:14 DEBUG }, 04:38:14 DEBUG "value": [ 04:38:14 DEBUG 1713584294, 04:38:14 DEBUG "0.011449872994891237" 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG ] 04:38:14 DEBUG } 04:38:14 DEBUG } 04:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584295" 04:38:15 INFO [http_cmd]: http status code OK 04:38:15 DEBUG --- status code --- 04:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:15 DEBUG --- http response --- 04:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:15 DEBUG ----- output ----- 04:38:15 DEBUG { 04:38:15 DEBUG "status": "success", 04:38:15 DEBUG "data": { 04:38:15 DEBUG "resultType": "vector", 04:38:15 DEBUG "result": [ 04:38:15 DEBUG { 04:38:15 DEBUG "metric": { 04:38:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:38:15 DEBUG }, 04:38:15 DEBUG "value": [ 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG [...] 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:38:15 DEBUG }, 04:38:15 DEBUG "value": [ 04:38:15 DEBUG 1713584295, 04:38:15 DEBUG "5111730176" 04:38:15 DEBUG ] 04:38:15 DEBUG } 04:38:15 DEBUG ] 04:38:15 DEBUG } 04:38:15 DEBUG } 04:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584295" 04:38:15 INFO [http_cmd]: http status code OK 04:38:15 DEBUG --- status code --- 04:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:15 DEBUG --- http response --- 04:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:15 DEBUG ----- output ----- 04:38:15 DEBUG { 04:38:15 DEBUG "status": "success", 04:38:15 DEBUG "data": { 04:38:15 DEBUG "resultType": "vector", 04:38:15 DEBUG "result": [ 04:38:15 DEBUG { 04:38:15 DEBUG "metric": { 04:38:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:38:15 DEBUG }, 04:38:15 DEBUG "value": [ 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG [...] 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:38:15 DEBUG }, 04:38:15 DEBUG "value": [ 04:38:15 DEBUG 1713584295, 04:38:15 DEBUG "1.0829494168154687" 04:38:15 DEBUG ] 04:38:15 DEBUG } 04:38:15 DEBUG ] 04:38:15 DEBUG } 04:38:15 DEBUG } 04:38:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584295" 04:38:15 INFO [http_cmd]: http status code OK 04:38:15 DEBUG --- status code --- 04:38:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:38:15 DEBUG --- http response --- 04:38:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:38:15 DEBUG ----- output ----- 04:38:15 DEBUG { 04:38:15 DEBUG "status": "success", 04:38:15 DEBUG "data": { 04:38:15 DEBUG "resultType": "vector", 04:38:15 DEBUG "result": [ 04:38:15 DEBUG { 04:38:15 DEBUG "metric": { 04:38:15 DEBUG "instance": "10.142.0.107:9100" 04:38:15 DEBUG }, 04:38:15 DEBUG "value": [ 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG [...] 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG 04:38:15 DEBUG "instance": "10.142.0.43:9100" 04:38:15 DEBUG }, 04:38:15 DEBUG "value": [ 04:38:15 DEBUG 1713584295, 04:38:15 DEBUG "0" 04:38:15 DEBUG ] 04:38:15 DEBUG } 04:38:15 DEBUG ] 04:38:15 DEBUG } 04:38:15 DEBUG } 04:38:31 INFO 04:38:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:38:31 INFO [loop_until]: OK (rc = 0) 04:38:31 DEBUG --- stdout --- 04:38:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8594Mi am-767f6b67b5-qcqz2 9m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 16m 10570Mi ds-cts-1 13m 10678Mi ds-cts-2 12m 10601Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 13m 3631Mi ds-idrepo-2 33m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1002m 5295Mi 04:38:31 DEBUG --- stderr --- 04:38:31 DEBUG 04:38:37 INFO 04:38:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:38:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:38:37 INFO [loop_until]: OK (rc = 0) 04:38:37 DEBUG --- stdout --- 04:38:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6277Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 7372Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 11296Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 13685Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5189Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 11530Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 10791Mi 9% gke-xlou-cdm-ds-0f601751-j488 85m 0% 6806Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 15562Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 94m 0% 7499Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 67m 0% 13192Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 9301Mi 15% 04:38:37 DEBUG --- stderr --- 04:38:37 DEBUG 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 ERROR [http_cmd]: ERROR 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 ERROR --- status code --- 04:39:14 DEBUG --- status code --- 04:39:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 ERROR --- http response --- 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:39:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-idrepo-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-idrepo-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-idrepo-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-idrepo-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-cts-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-cts-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-idrepo-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-idrepo-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-cts-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0.085" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "instance": "10.142.0.107:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "instance": "10.142.0.43:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "28196.36666666667" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "instance": "10.142.0.107:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "instance": "10.142.0.43:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "28518.3" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584354" 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "instance": "10.142.0.107:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "instance": "10.142.0.43:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0.0003000000000137485" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG { 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-cts-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:14 DEBUG 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG 04:39:14 DEBUG { 04:39:14 DEBUG "pod": "ds-cts-2" 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG }, 04:39:14 DEBUG "data": { 04:39:14 DEBUG "value": [ 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "result": [ 04:39:14 DEBUG "0" 04:39:14 DEBUG { 04:39:14 DEBUG ] 04:39:14 DEBUG "metric": { 04:39:14 DEBUG } 04:39:14 DEBUG "instance": "10.142.0.107:9100" 04:39:14 DEBUG ] 04:39:14 DEBUG }, 04:39:14 DEBUG } 04:39:14 DEBUG "value": [ 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG } 04:39:14 DEBUG 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "instance": "10.142.0.43:9100" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "3036652053504" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-idrepo-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-cts-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "ds-cts-0" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-cts-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584354" 04:39:14 INFO [http_cmd]: http status code OK 04:39:14 DEBUG --- status code --- 04:39:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:14 DEBUG --- http response --- 04:39:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:14 DEBUG ----- output ----- 04:39:14 DEBUG { 04:39:14 DEBUG "status": "success", 04:39:14 DEBUG "data": { 04:39:14 DEBUG "resultType": "vector", 04:39:14 DEBUG "result": [ 04:39:14 DEBUG { 04:39:14 DEBUG "metric": { 04:39:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG [...] 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG 04:39:14 DEBUG "pod": "ds-cts-2" 04:39:14 DEBUG }, 04:39:14 DEBUG "value": [ 04:39:14 DEBUG 1713584354, 04:39:14 DEBUG "0" 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG ] 04:39:14 DEBUG } 04:39:14 DEBUG } 04:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584355" 04:39:15 INFO [http_cmd]: http status code OK 04:39:15 DEBUG --- status code --- 04:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:15 DEBUG --- http response --- 04:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:15 DEBUG ----- output ----- 04:39:15 DEBUG { 04:39:15 DEBUG "status": "success", 04:39:15 DEBUG "data": { 04:39:15 DEBUG "resultType": "vector", 04:39:15 DEBUG "result": [ 04:39:15 DEBUG { 04:39:15 DEBUG "metric": { 04:39:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:39:15 DEBUG }, 04:39:15 DEBUG "value": [ 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG [...] 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:39:15 DEBUG }, 04:39:15 DEBUG "value": [ 04:39:15 DEBUG 1713584355, 04:39:15 DEBUG "5848961024" 04:39:15 DEBUG ] 04:39:15 DEBUG } 04:39:15 DEBUG ] 04:39:15 DEBUG } 04:39:15 DEBUG } 04:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584355" 04:39:15 INFO [http_cmd]: http status code OK 04:39:15 DEBUG --- status code --- 04:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:15 DEBUG --- http response --- 04:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:15 DEBUG ----- output ----- 04:39:15 DEBUG { 04:39:15 DEBUG "status": "success", 04:39:15 DEBUG "data": { 04:39:15 DEBUG "resultType": "vector", 04:39:15 DEBUG "result": [ 04:39:15 DEBUG { 04:39:15 DEBUG "metric": { 04:39:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:39:15 DEBUG }, 04:39:15 DEBUG "value": [ 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG [...] 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:39:15 DEBUG }, 04:39:15 DEBUG "value": [ 04:39:15 DEBUG 1713584355, 04:39:15 DEBUG "1.0303144049226756" 04:39:15 DEBUG ] 04:39:15 DEBUG } 04:39:15 DEBUG ] 04:39:15 DEBUG } 04:39:15 DEBUG } 04:39:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584355" 04:39:15 INFO [http_cmd]: http status code OK 04:39:15 DEBUG --- status code --- 04:39:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:39:15 DEBUG --- http response --- 04:39:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:39:15 DEBUG ----- output ----- 04:39:15 DEBUG { 04:39:15 DEBUG "status": "success", 04:39:15 DEBUG "data": { 04:39:15 DEBUG "resultType": "vector", 04:39:15 DEBUG "result": [ 04:39:15 DEBUG { 04:39:15 DEBUG "metric": { 04:39:15 DEBUG "instance": "10.142.0.107:9100" 04:39:15 DEBUG }, 04:39:15 DEBUG "value": [ 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG [...] 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG 04:39:15 DEBUG "instance": "10.142.0.43:9100" 04:39:15 DEBUG }, 04:39:15 DEBUG "value": [ 04:39:15 DEBUG 1713584355, 04:39:15 DEBUG "0" 04:39:15 DEBUG ] 04:39:15 DEBUG } 04:39:15 DEBUG ] 04:39:15 DEBUG } 04:39:15 DEBUG } 04:39:31 INFO 04:39:31 INFO [loop_until]: kubectl --namespace=xlou top pods 04:39:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:39:32 INFO [loop_until]: OK (rc = 0) 04:39:32 DEBUG --- stdout --- 04:39:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8673Mi am-767f6b67b5-vbltz 6m 8600Mi ds-cts-0 13m 10571Mi ds-cts-1 14m 10678Mi ds-cts-2 12m 10601Mi ds-idrepo-0 14m 3722Mi ds-idrepo-1 13m 3631Mi ds-idrepo-2 12m 3527Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1016m 6080Mi 04:39:32 DEBUG --- stderr --- 04:39:32 DEBUG 04:39:37 INFO 04:39:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:39:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:39:37 INFO [loop_until]: OK (rc = 0) 04:39:37 DEBUG --- stdout --- 04:39:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 6276Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7361Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 11288Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 13701Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5180Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11517Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 10788Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6811Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 15564Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 97m 0% 7498Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 13186Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1042m 6% 9949Mi 16% 04:39:37 DEBUG --- stderr --- 04:39:37 DEBUG 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 ERROR [http_cmd]: ERROR 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 ERROR --- status code --- 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG --- status code --- 04:40:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 ERROR --- http response --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:40:14 DEBUG --- http response --- 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:40:14 DEBUG "data": { 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "result": [ 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "data": { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "ds-idrepo-0" 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG "pod": "ds-idrepo-0" 04:40:14 DEBUG }, 04:40:14 DEBUG "result": [ 04:40:14 DEBUG }, 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG { 04:40:14 DEBUG "value": [ 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG "metric": { 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-0" 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG }, 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG "value": [ 04:40:14 DEBUG [...] 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-idrepo-2" 04:40:14 DEBUG [...] 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG "pod": "ds-idrepo-2" 04:40:14 DEBUG }, 04:40:14 DEBUG 04:40:14 DEBUG }, 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG "value": [ 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG "0" 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG }, 04:40:14 DEBUG ] 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG "value": [ 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG ] 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG "0" 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG ] 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG } 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG } 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "ds-idrepo-0" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-idrepo-2" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "351368824.61549157" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0.085" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "instance": "10.142.0.107:9100" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "instance": "10.142.0.43:9100" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "27267.166666666668" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "instance": "10.142.0.107:9100" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "instance": "10.142.0.43:9100" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "26495.566666666666" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584414" 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 DEBUG ----- output ----- 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG "data": { 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG "result": [ 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "ds-idrepo-0" 04:40:14 DEBUG "instance": "10.142.0.107:9100" 04:40:14 DEBUG "data": { 04:40:14 DEBUG "pod": "ds-cts-0" 04:40:14 DEBUG }, 04:40:14 DEBUG }, 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG "value": [ 04:40:14 DEBUG "result": [ 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG { 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "metric": { 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-0" 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG [...] 04:40:14 DEBUG }, 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG "instance": "10.142.0.43:9100" 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG }, 04:40:14 DEBUG }, 04:40:14 DEBUG [...] 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG "0.00030000000002038026" 04:40:14 DEBUG 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG ] 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG }, 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG ] 04:40:14 DEBUG "value": [ 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 DEBUG "0" 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "instance": "10.142.0.107:9100" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "instance": "10.142.0.43:9100" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "3036652319744" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584414" 04:40:14 INFO [http_cmd]: http status code OK 04:40:14 DEBUG --- status code --- 04:40:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:14 DEBUG --- http response --- 04:40:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:14 DEBUG ----- output ----- 04:40:14 DEBUG { 04:40:14 DEBUG "status": "success", 04:40:14 DEBUG "data": { 04:40:14 DEBUG "resultType": "vector", 04:40:14 DEBUG "result": [ 04:40:14 DEBUG { 04:40:14 DEBUG "metric": { 04:40:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG [...] 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG 04:40:14 DEBUG "pod": "ds-cts-2" 04:40:14 DEBUG }, 04:40:14 DEBUG "value": [ 04:40:14 DEBUG 1713584414, 04:40:14 DEBUG "0.004427931874045213" 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG ] 04:40:14 DEBUG } 04:40:14 DEBUG } 04:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584415" 04:40:15 INFO [http_cmd]: http status code OK 04:40:15 DEBUG --- status code --- 04:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:15 DEBUG --- http response --- 04:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:15 DEBUG ----- output ----- 04:40:15 DEBUG { 04:40:15 DEBUG "status": "success", 04:40:15 DEBUG "data": { 04:40:15 DEBUG "resultType": "vector", 04:40:15 DEBUG "result": [ 04:40:15 DEBUG { 04:40:15 DEBUG "metric": { 04:40:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:40:15 DEBUG }, 04:40:15 DEBUG "value": [ 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG [...] 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:40:15 DEBUG }, 04:40:15 DEBUG "value": [ 04:40:15 DEBUG 1713584415, 04:40:15 DEBUG "6472896512" 04:40:15 DEBUG ] 04:40:15 DEBUG } 04:40:15 DEBUG ] 04:40:15 DEBUG } 04:40:15 DEBUG } 04:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584415" 04:40:15 INFO [http_cmd]: http status code OK 04:40:15 DEBUG --- status code --- 04:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:15 DEBUG --- http response --- 04:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:15 DEBUG ----- output ----- 04:40:15 DEBUG { 04:40:15 DEBUG "status": "success", 04:40:15 DEBUG "data": { 04:40:15 DEBUG "resultType": "vector", 04:40:15 DEBUG "result": [ 04:40:15 DEBUG { 04:40:15 DEBUG "metric": { 04:40:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:40:15 DEBUG }, 04:40:15 DEBUG "value": [ 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG [...] 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:40:15 DEBUG }, 04:40:15 DEBUG "value": [ 04:40:15 DEBUG 1713584415, 04:40:15 DEBUG "1.0281714522493604" 04:40:15 DEBUG ] 04:40:15 DEBUG } 04:40:15 DEBUG ] 04:40:15 DEBUG } 04:40:15 DEBUG } 04:40:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584415" 04:40:15 INFO [http_cmd]: http status code OK 04:40:15 DEBUG --- status code --- 04:40:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:40:15 DEBUG --- http response --- 04:40:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:40:15 DEBUG ----- output ----- 04:40:15 DEBUG { 04:40:15 DEBUG "status": "success", 04:40:15 DEBUG "data": { 04:40:15 DEBUG "resultType": "vector", 04:40:15 DEBUG "result": [ 04:40:15 DEBUG { 04:40:15 DEBUG "metric": { 04:40:15 DEBUG "instance": "10.142.0.107:9100" 04:40:15 DEBUG }, 04:40:15 DEBUG "value": [ 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG [...] 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG 04:40:15 DEBUG "instance": "10.142.0.43:9100" 04:40:15 DEBUG }, 04:40:15 DEBUG "value": [ 04:40:15 DEBUG 1713584415, 04:40:15 DEBUG "0.0003333333333330302" 04:40:15 DEBUG ] 04:40:15 DEBUG } 04:40:15 DEBUG ] 04:40:15 DEBUG } 04:40:15 DEBUG } 04:40:32 INFO 04:40:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:40:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:40:32 INFO [loop_until]: OK (rc = 0) 04:40:32 DEBUG --- stdout --- 04:40:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 6m 8601Mi ds-cts-0 11m 10570Mi ds-cts-1 15m 10677Mi ds-cts-2 10m 10600Mi ds-idrepo-0 16m 3722Mi ds-idrepo-1 12m 3631Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 7m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 4m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 998m 6599Mi 04:40:32 DEBUG --- stderr --- 04:40:32 DEBUG 04:40:37 INFO 04:40:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:40:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:40:37 INFO [loop_until]: OK (rc = 0) 04:40:37 DEBUG --- stdout --- 04:40:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 83m 0% 6287Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 77m 0% 7365Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 11287Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 13693Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5195Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 69m 0% 11518Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 10791Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 6801Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 15566Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 7488Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 13188Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 10485Mi 17% 04:40:37 DEBUG --- stderr --- 04:40:37 DEBUG 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "ds-idrepo-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-idrepo-2" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 ERROR [http_cmd]: ERROR 04:41:14 ERROR --- status code --- 04:41:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:41:14 ERROR --- http response --- 04:41:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:41:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "ds-idrepo-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-idrepo-2" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "ds-cts-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-cts-2" 04:41:14 DEBUG }, 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "ds-idrepo-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-idrepo-2" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-idrepo-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-cts-2" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0.086" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "instance": "10.142.0.107:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "instance": "10.142.0.43:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "26309.86666666667" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "instance": "10.142.0.107:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "instance": "10.142.0.43:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "27975.166666666664" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "instance": "10.142.0.107:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "instance": "10.142.0.43:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "3036652696576" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:14 DEBUG { 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG { 04:41:14 DEBUG "data": { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "result": [ 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG { 04:41:14 DEBUG "result": [ 04:41:14 DEBUG "metric": { 04:41:14 DEBUG { 04:41:14 DEBUG "pod": "ds-cts-0" 04:41:14 DEBUG "metric": { 04:41:14 DEBUG }, 04:41:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:41:14 DEBUG "value": [ 04:41:14 DEBUG }, 04:41:14 DEBUG 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-cts-2" 04:41:14 DEBUG 04:41:14 DEBUG }, 04:41:14 DEBUG "pod": "ds-idrepo-0" 04:41:14 DEBUG "value": [ 04:41:14 DEBUG }, 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG "0" 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG ] 04:41:14 DEBUG "2013.1940886699508" 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "ds-idrepo-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG [...] 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "pod": "ds-cts-2" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584474" 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 INFO [http_cmd]: http status code OK 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG --- status code --- 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG --- http response --- 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG { 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG "data": { 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG "result": [ 04:41:14 DEBUG { 04:41:14 DEBUG "metric": { 04:41:14 DEBUG "pod": "ds-cts-0" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:14 DEBUG 04:41:14 DEBUG ----- output ----- 04:41:14 DEBUG 04:41:14 DEBUG { 04:41:14 DEBUG 04:41:14 DEBUG "status": "success", 04:41:14 DEBUG [...] 04:41:14 DEBUG "data": { 04:41:14 DEBUG 04:41:14 DEBUG "resultType": "vector", 04:41:14 DEBUG 04:41:14 DEBUG "result": [ 04:41:14 DEBUG 04:41:14 DEBUG { 04:41:14 DEBUG "pod": "ds-cts-2" 04:41:14 DEBUG "metric": { 04:41:14 DEBUG }, 04:41:14 DEBUG "instance": "10.142.0.107:9100" 04:41:14 DEBUG "value": [ 04:41:14 DEBUG }, 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG "0" 04:41:14 DEBUG 04:41:14 DEBUG ] 04:41:14 DEBUG 04:41:14 DEBUG } 04:41:14 DEBUG 04:41:14 DEBUG ] 04:41:14 DEBUG [...] 04:41:14 DEBUG } 04:41:14 DEBUG 04:41:14 DEBUG } 04:41:14 DEBUG 04:41:14 DEBUG 04:41:14 DEBUG "instance": "10.142.0.43:9100" 04:41:14 DEBUG }, 04:41:14 DEBUG "value": [ 04:41:14 DEBUG 1713584474, 04:41:14 DEBUG "0.0003333333333406093" 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG ] 04:41:14 DEBUG } 04:41:14 DEBUG } 04:41:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584474" 04:41:15 INFO [http_cmd]: http status code OK 04:41:15 DEBUG --- status code --- 04:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:15 DEBUG --- http response --- 04:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:41:15 DEBUG ----- output ----- 04:41:15 DEBUG { 04:41:15 DEBUG "status": "success", 04:41:15 DEBUG "data": { 04:41:15 DEBUG "resultType": "vector", 04:41:15 DEBUG "result": [ 04:41:15 DEBUG { 04:41:15 DEBUG "metric": { 04:41:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG [...] 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG "pod": "ds-cts-2" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 1713584474, 04:41:15 DEBUG "0" 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG } 04:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584475" 04:41:15 INFO [http_cmd]: http status code OK 04:41:15 DEBUG --- status code --- 04:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:15 DEBUG --- http response --- 04:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:15 DEBUG ----- output ----- 04:41:15 DEBUG { 04:41:15 DEBUG "status": "success", 04:41:15 DEBUG "data": { 04:41:15 DEBUG "resultType": "vector", 04:41:15 DEBUG "result": [ 04:41:15 DEBUG { 04:41:15 DEBUG "metric": { 04:41:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG [...] 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 1713584475, 04:41:15 DEBUG "7196672000" 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG } 04:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584475" 04:41:15 INFO [http_cmd]: http status code OK 04:41:15 DEBUG --- status code --- 04:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:15 DEBUG --- http response --- 04:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:15 DEBUG ----- output ----- 04:41:15 DEBUG { 04:41:15 DEBUG "status": "success", 04:41:15 DEBUG "data": { 04:41:15 DEBUG "resultType": "vector", 04:41:15 DEBUG "result": [ 04:41:15 DEBUG { 04:41:15 DEBUG "metric": { 04:41:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG [...] 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 1713584475, 04:41:15 DEBUG "1.0354466846240131" 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG } 04:41:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584475" 04:41:15 INFO [http_cmd]: http status code OK 04:41:15 DEBUG --- status code --- 04:41:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:41:15 DEBUG --- http response --- 04:41:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:41:15 DEBUG ----- output ----- 04:41:15 DEBUG { 04:41:15 DEBUG "status": "success", 04:41:15 DEBUG "data": { 04:41:15 DEBUG "resultType": "vector", 04:41:15 DEBUG "result": [ 04:41:15 DEBUG { 04:41:15 DEBUG "metric": { 04:41:15 DEBUG "instance": "10.142.0.107:9100" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG [...] 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG 04:41:15 DEBUG "instance": "10.142.0.43:9100" 04:41:15 DEBUG }, 04:41:15 DEBUG "value": [ 04:41:15 DEBUG 1713584475, 04:41:15 DEBUG "0.0003333333333330302" 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG ] 04:41:15 DEBUG } 04:41:15 DEBUG } 04:41:32 INFO 04:41:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:41:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:41:32 INFO [loop_until]: OK (rc = 0) 04:41:32 DEBUG --- stdout --- 04:41:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8594Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 14m 10570Mi ds-cts-1 13m 10676Mi ds-cts-2 12m 10600Mi ds-idrepo-0 14m 3720Mi ds-idrepo-1 35m 3631Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 5m 1235Mi lodemon-df4b7b554-2gxfx 5m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1021m 7412Mi 04:41:32 DEBUG --- stderr --- 04:41:32 DEBUG 04:41:37 INFO 04:41:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:41:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:41:37 INFO [loop_until]: OK (rc = 0) 04:41:37 DEBUG --- stdout --- 04:41:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 79m 0% 6273Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 7362Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11281Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 13696Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5181Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 11518Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 2932Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 10795Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 6808Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 15571Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 113m 0% 7502Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 13194Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1083m 6% 11140Mi 18% 04:41:37 DEBUG --- stderr --- 04:41:37 DEBUG 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "ds-idrepo-0" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "ds-idrepo-2" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "ds-cts-0" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "ds-cts-2" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 ERROR [http_cmd]: ERROR 04:42:14 ERROR --- status code --- 04:42:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:42:14 ERROR --- http response --- 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:42:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "ds-idrepo-0" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "ds-idrepo-2" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "ds-idrepo-0" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "ds-idrepo-2" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "464442853.3085238" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "0.086" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584534" 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "instance": "10.142.0.107:9100" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "instance": "10.142.0.43:9100" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "26020.7" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "instance": "10.142.0.107:9100" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "instance": "10.142.0.43:9100" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "24349.766666666666" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584534" 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:14 INFO [http_cmd]: http status code OK 04:42:14 DEBUG --- status code --- 04:42:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:14 DEBUG --- http response --- 04:42:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:14 DEBUG ----- output ----- 04:42:14 DEBUG { 04:42:14 DEBUG "status": "success", 04:42:14 DEBUG "data": { 04:42:14 DEBUG "resultType": "vector", 04:42:14 DEBUG "result": [ 04:42:14 DEBUG { 04:42:14 DEBUG "metric": { 04:42:14 DEBUG "instance": "10.142.0.107:9100" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG [...] 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG 04:42:14 DEBUG "instance": "10.142.0.43:9100" 04:42:14 DEBUG }, 04:42:14 DEBUG "value": [ 04:42:14 DEBUG 1713584534, 04:42:14 DEBUG "3036653157376" 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG ] 04:42:14 DEBUG } 04:42:14 DEBUG } 04:42:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584534" 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "pod": "ds-idrepo-0" 04:42:15 DEBUG "pod": "ds-cts-0" 04:42:15 DEBUG }, 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG [...] 04:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584535" 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "pod": "ds-cts-2" 04:42:15 DEBUG "pod": "ds-cts-2" 04:42:15 DEBUG }, 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584534, 04:42:15 DEBUG 1713584534, 04:42:15 DEBUG "0" 04:42:15 DEBUG "0" 04:42:15 DEBUG ] 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584534, 04:42:15 DEBUG "746.767460596735" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "pod": "ds-cts-0" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "pod": "ds-cts-2" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584535, 04:42:15 DEBUG "0" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584535" 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "instance": "10.142.0.107:9100" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "instance": "10.142.0.43:9100" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584535, 04:42:15 DEBUG "0.00019999999988537336" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584535" 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584535, 04:42:15 DEBUG "1.260933180229353" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584535" 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584535, 04:42:15 DEBUG "7870812160" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584535" 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584535, 04:42:15 DEBUG "1.0289036416183053" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584535" 04:42:15 INFO [http_cmd]: http status code OK 04:42:15 DEBUG --- status code --- 04:42:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:42:15 DEBUG --- http response --- 04:42:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:42:15 DEBUG ----- output ----- 04:42:15 DEBUG { 04:42:15 DEBUG "status": "success", 04:42:15 DEBUG "data": { 04:42:15 DEBUG "resultType": "vector", 04:42:15 DEBUG "result": [ 04:42:15 DEBUG { 04:42:15 DEBUG "metric": { 04:42:15 DEBUG "instance": "10.142.0.107:9100" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG [...] 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG 04:42:15 DEBUG "instance": "10.142.0.43:9100" 04:42:15 DEBUG }, 04:42:15 DEBUG "value": [ 04:42:15 DEBUG 1713584535, 04:42:15 DEBUG "0" 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG ] 04:42:15 DEBUG } 04:42:15 DEBUG } 04:42:32 INFO 04:42:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:42:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:42:32 INFO [loop_until]: OK (rc = 0) 04:42:32 DEBUG --- stdout --- 04:42:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8594Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 8m 8601Mi ds-cts-0 12m 10570Mi ds-cts-1 13m 10677Mi ds-cts-2 11m 10602Mi ds-idrepo-0 14m 3720Mi ds-idrepo-1 11m 3631Mi ds-idrepo-2 12m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 927m 7301Mi 04:42:32 DEBUG --- stderr --- 04:42:32 DEBUG 04:42:37 INFO 04:42:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:42:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:42:37 INFO [loop_until]: OK (rc = 0) 04:42:37 DEBUG --- stdout --- 04:42:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6287Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 7350Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11284Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 72m 0% 13686Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11516Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 82m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 10800Mi 9% gke-xlou-cdm-ds-0f601751-j488 68m 0% 6805Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 15559Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 7500Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13192Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 784m 4% 11144Mi 19% 04:42:37 DEBUG --- stderr --- 04:42:37 DEBUG 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "ds-idrepo-0" 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "ds-idrepo-2" 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "ds-cts-0" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "ds-cts-2" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 ERROR [http_cmd]: ERROR 04:43:14 ERROR --- status code --- 04:43:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:43:14 ERROR --- http response --- 04:43:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:43:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "ds-idrepo-0" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "ds-idrepo-2" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "ds-idrepo-0" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "ds-idrepo-2" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG { 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "ds-cts-2" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "ds-cts-2" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "21333.98407673861" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "0.086" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584594" 04:43:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584594" 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "instance": "10.142.0.107:9100" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "instance": "10.142.0.43:9100" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "26141.233333333334" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:14 INFO [http_cmd]: http status code OK 04:43:14 DEBUG --- status code --- 04:43:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:14 DEBUG --- http response --- 04:43:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:14 DEBUG ----- output ----- 04:43:14 DEBUG { 04:43:14 DEBUG "status": "success", 04:43:14 DEBUG "data": { 04:43:14 DEBUG "resultType": "vector", 04:43:14 DEBUG "result": [ 04:43:14 DEBUG { 04:43:14 DEBUG "metric": { 04:43:14 DEBUG "instance": "10.142.0.107:9100" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG [...] 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG 04:43:14 DEBUG "instance": "10.142.0.43:9100" 04:43:14 DEBUG }, 04:43:14 DEBUG "value": [ 04:43:14 DEBUG 1713584594, 04:43:14 DEBUG "26452.033333333333" 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG ] 04:43:14 DEBUG } 04:43:14 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584595" 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "pod": "ds-idrepo-0" 04:43:15 DEBUG }, 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584595" 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "pod": "ds-cts-2" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "instance": "10.142.0.107:9100" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "instance": "10.142.0.43:9100" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "3036653419520" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG { 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "pod": "ds-cts-0" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "pod": "ds-cts-2" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "pod": "ds-cts-2" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "pod": "ds-cts-0" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "pod": "ds-cts-2" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "instance": "10.142.0.107:9100" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "instance": "10.142.0.43:9100" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0.00043333333335435025" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "pod": "ds-cts-2" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0.004803161210519313" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "7753322496" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0.9738256087378298" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584595" 04:43:15 INFO [http_cmd]: http status code OK 04:43:15 DEBUG --- status code --- 04:43:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:43:15 DEBUG --- http response --- 04:43:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:43:15 DEBUG ----- output ----- 04:43:15 DEBUG { 04:43:15 DEBUG "status": "success", 04:43:15 DEBUG "data": { 04:43:15 DEBUG "resultType": "vector", 04:43:15 DEBUG "result": [ 04:43:15 DEBUG { 04:43:15 DEBUG "metric": { 04:43:15 DEBUG "instance": "10.142.0.107:9100" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG [...] 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG 04:43:15 DEBUG "instance": "10.142.0.43:9100" 04:43:15 DEBUG }, 04:43:15 DEBUG "value": [ 04:43:15 DEBUG 1713584595, 04:43:15 DEBUG "0" 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG ] 04:43:15 DEBUG } 04:43:15 DEBUG } 04:43:32 INFO 04:43:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:43:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:43:32 INFO [loop_until]: OK (rc = 0) 04:43:32 DEBUG --- stdout --- 04:43:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 13m 10572Mi ds-cts-1 12m 10676Mi ds-cts-2 13m 10600Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 13m 3631Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1006m 7315Mi 04:43:32 DEBUG --- stderr --- 04:43:32 DEBUG 04:43:37 INFO 04:43:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:43:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:43:37 INFO [loop_until]: OK (rc = 0) 04:43:37 DEBUG --- stdout --- 04:43:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6287Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7346Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 11276Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 13704Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11506Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 2929Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 10797Mi 9% gke-xlou-cdm-ds-0f601751-j488 68m 0% 6811Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15568Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 96m 0% 7497Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13199Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1041m 6% 11158Mi 19% 04:43:37 DEBUG --- stderr --- 04:43:37 DEBUG 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "ds-idrepo-0" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "ds-idrepo-2" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584654" 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 ERROR [http_cmd]: ERROR 04:44:14 ERROR --- status code --- 04:44:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:44:14 ERROR --- http response --- 04:44:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:44:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG }, 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG }, 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "ds-cts-0" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "ds-cts-2" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "ds-idrepo-0" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "ds-idrepo-2" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "ds-idrepo-0" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "ds-idrepo-2" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "ds-idrepo-0" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "ds-idrepo-0" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "152739324.2615427" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "0.086" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584654" 04:44:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584654" 04:44:14 INFO [http_cmd]: http status code OK 04:44:14 DEBUG --- status code --- 04:44:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:14 DEBUG --- http response --- 04:44:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:14 DEBUG ----- output ----- 04:44:14 DEBUG { 04:44:14 DEBUG "status": "success", 04:44:14 DEBUG "data": { 04:44:14 DEBUG "resultType": "vector", 04:44:14 DEBUG "result": [ 04:44:14 DEBUG { 04:44:14 DEBUG "metric": { 04:44:14 DEBUG "instance": "10.142.0.107:9100" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG [...] 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG 04:44:14 DEBUG "instance": "10.142.0.43:9100" 04:44:14 DEBUG }, 04:44:14 DEBUG "value": [ 04:44:14 DEBUG 1713584654, 04:44:14 DEBUG "26214.199999999997" 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG ] 04:44:14 DEBUG } 04:44:14 DEBUG } 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "instance": "10.142.0.107:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "instance": "10.142.0.43:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584654, 04:44:15 DEBUG "27102.4" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584655" 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "pod": "ds-cts-0" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "pod": "ds-cts-2" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "0" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584655" 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584655" 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584655" 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG { 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:44:15 DEBUG "data": { 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "pod": "ds-cts-0" 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG }, 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "value": [ 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584655" 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "pod": "ds-idrepo-0" 04:44:15 DEBUG 04:44:15 DEBUG "instance": "10.142.0.107:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG [...] 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG "pod": "ds-cts-2" 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG }, 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG 04:44:15 DEBUG "pod": "ds-cts-2" 04:44:15 DEBUG "0" 04:44:15 DEBUG "instance": "10.142.0.43:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG ] 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG } 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG ] 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "0" 04:44:15 DEBUG } 04:44:15 DEBUG "3036653763584" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "pod": "ds-idrepo-0" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "2732.1039582761464" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584655" 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "instance": "10.142.0.107:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "instance": "10.142.0.43:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "0.00033333333334724103" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584655" 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "pod": "ds-idrepo-0" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "0.008160578790164614" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584655" 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "7769264128" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584655" 04:44:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584655" 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "1.0286353845285259" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:15 INFO [http_cmd]: http status code OK 04:44:15 DEBUG --- status code --- 04:44:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:44:15 DEBUG --- http response --- 04:44:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:44:15 DEBUG ----- output ----- 04:44:15 DEBUG { 04:44:15 DEBUG "status": "success", 04:44:15 DEBUG "data": { 04:44:15 DEBUG "resultType": "vector", 04:44:15 DEBUG "result": [ 04:44:15 DEBUG { 04:44:15 DEBUG "metric": { 04:44:15 DEBUG "instance": "10.142.0.107:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG [...] 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG 04:44:15 DEBUG "instance": "10.142.0.43:9100" 04:44:15 DEBUG }, 04:44:15 DEBUG "value": [ 04:44:15 DEBUG 1713584655, 04:44:15 DEBUG "0.0003333333333330302" 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG ] 04:44:15 DEBUG } 04:44:15 DEBUG } 04:44:32 INFO 04:44:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:44:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:44:32 INFO [loop_until]: OK (rc = 0) 04:44:32 DEBUG --- stdout --- 04:44:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8594Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 13m 10570Mi ds-cts-1 12m 10676Mi ds-cts-2 11m 10601Mi ds-idrepo-0 38m 3721Mi ds-idrepo-1 13m 3632Mi ds-idrepo-2 12m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1020m 7315Mi 04:44:32 DEBUG --- stderr --- 04:44:32 DEBUG 04:44:37 INFO 04:44:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:44:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:44:37 INFO [loop_until]: OK (rc = 0) 04:44:37 DEBUG --- stdout --- 04:44:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 92m 0% 7354Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 11278Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 13701Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 11515Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 10798Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 6827Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 15565Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 96m 0% 7491Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 67m 0% 13198Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 11159Mi 19% 04:44:37 DEBUG --- stderr --- 04:44:37 DEBUG 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "ds-idrepo-0" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "ds-idrepo-2" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584714" 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 ERROR [http_cmd]: ERROR 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 ERROR --- status code --- 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG --- status code --- 04:45:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 ERROR --- http response --- 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:14 DEBUG }, 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:14 DEBUG }, 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "ds-cts-0" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "ds-cts-2" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "ds-idrepo-0" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "ds-idrepo-2" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "ds-idrepo-0" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "ds-idrepo-2" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "27353427.207870685" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:14 INFO [http_cmd]: http status code OK 04:45:14 DEBUG --- status code --- 04:45:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:14 DEBUG --- http response --- 04:45:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:14 DEBUG ----- output ----- 04:45:14 DEBUG { 04:45:14 DEBUG "status": "success", 04:45:14 DEBUG "data": { 04:45:14 DEBUG "resultType": "vector", 04:45:14 DEBUG "result": [ 04:45:14 DEBUG { 04:45:14 DEBUG "metric": { 04:45:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG [...] 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG 04:45:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:14 DEBUG }, 04:45:14 DEBUG "value": [ 04:45:14 DEBUG 1713584714, 04:45:14 DEBUG "0" 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG ] 04:45:14 DEBUG } 04:45:14 DEBUG } 04:45:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584714" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "pod": "ds-cts-2" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584714, 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0.086" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584715" 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "instance": "10.142.0.107:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "instance": "10.142.0.43:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "24498.8" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "instance": "10.142.0.107:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "instance": "10.142.0.43:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "26017.5" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584715" 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584715" 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584715" 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "pod": "ds-cts-0" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "pod": "ds-cts-2" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG --- http response --- 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584715" 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "instance": "10.142.0.107:9100" 04:45:15 DEBUG "pod": "ds-cts-0" 04:45:15 DEBUG }, 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "instance": "10.142.0.43:9100" 04:45:15 DEBUG "pod": "ds-cts-2" 04:45:15 DEBUG }, 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "3036654081024" 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "pod": "ds-idrepo-0" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "pod": "ds-cts-2" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "instance": "10.142.0.107:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "instance": "10.142.0.43:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0.0005333333332525096" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "pod": "ds-cts-2" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "7770263552" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584715" 04:45:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584715" 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "1.0300511034310862" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:15 INFO [http_cmd]: http status code OK 04:45:15 DEBUG --- status code --- 04:45:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:45:15 DEBUG --- http response --- 04:45:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:45:15 DEBUG ----- output ----- 04:45:15 DEBUG { 04:45:15 DEBUG "status": "success", 04:45:15 DEBUG "data": { 04:45:15 DEBUG "resultType": "vector", 04:45:15 DEBUG "result": [ 04:45:15 DEBUG { 04:45:15 DEBUG "metric": { 04:45:15 DEBUG "instance": "10.142.0.107:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG [...] 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG 04:45:15 DEBUG "instance": "10.142.0.43:9100" 04:45:15 DEBUG }, 04:45:15 DEBUG "value": [ 04:45:15 DEBUG 1713584715, 04:45:15 DEBUG "0" 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG ] 04:45:15 DEBUG } 04:45:15 DEBUG } 04:45:32 INFO 04:45:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:45:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:45:32 INFO [loop_until]: OK (rc = 0) 04:45:32 DEBUG --- stdout --- 04:45:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8673Mi am-767f6b67b5-vbltz 8m 8601Mi ds-cts-0 12m 10572Mi ds-cts-1 13m 10678Mi ds-cts-2 11m 10601Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 14m 3631Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 995m 7315Mi 04:45:32 DEBUG --- stderr --- 04:45:32 DEBUG 04:45:37 INFO 04:45:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:45:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:45:37 INFO [loop_until]: OK (rc = 0) 04:45:37 DEBUG --- stdout --- 04:45:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 6286Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7362Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 11285Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3347Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 13708Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5200Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 11518Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 2934Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 10788Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6815Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 15552Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 7510Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 65m 0% 13183Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 11159Mi 19% 04:45:37 DEBUG --- stderr --- 04:45:37 DEBUG 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584774" 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "ds-idrepo-0" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "ds-idrepo-2" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 ERROR [http_cmd]: ERROR 04:46:14 ERROR --- status code --- 04:46:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:46:14 ERROR --- http response --- 04:46:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:46:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "ds-idrepo-0" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "ds-idrepo-2" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "ds-cts-0" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "ds-cts-2" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "ds-idrepo-0" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "ds-idrepo-2" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584774" 04:46:14 INFO [http_cmd]: http status code OK 04:46:14 DEBUG --- status code --- 04:46:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:14 DEBUG --- http response --- 04:46:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:14 DEBUG ----- output ----- 04:46:14 DEBUG { 04:46:14 DEBUG "status": "success", 04:46:14 DEBUG "data": { 04:46:14 DEBUG "resultType": "vector", 04:46:14 DEBUG "result": [ 04:46:14 DEBUG { 04:46:14 DEBUG "metric": { 04:46:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG [...] 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG 04:46:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:14 DEBUG }, 04:46:14 DEBUG "value": [ 04:46:14 DEBUG 1713584774, 04:46:14 DEBUG "0" 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG ] 04:46:14 DEBUG } 04:46:14 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "34764604.0003566" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584775" 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "instance": "10.142.0.107:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "instance": "10.142.0.43:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "26096.13333333333" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0.086" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "instance": "10.142.0.107:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "instance": "10.142.0.43:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "24459.8" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "ds-cts-0" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "ds-cts-2" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "437.71732192208253" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "instance": "10.142.0.107:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "instance": "10.142.0.43:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "3036654390272" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "ds-cts-0" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "ds-cts-2" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "ds-idrepo-0" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "ds-cts-2" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "instance": "10.142.0.107:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "instance": "10.142.0.43:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0.00033333333334724103" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0.002028209590354034" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "7767601152" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "1.0307434649513714" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584775" 04:46:15 INFO [http_cmd]: http status code OK 04:46:15 DEBUG --- status code --- 04:46:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:46:15 DEBUG --- http response --- 04:46:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:46:15 DEBUG ----- output ----- 04:46:15 DEBUG { 04:46:15 DEBUG "status": "success", 04:46:15 DEBUG "data": { 04:46:15 DEBUG "resultType": "vector", 04:46:15 DEBUG "result": [ 04:46:15 DEBUG { 04:46:15 DEBUG "metric": { 04:46:15 DEBUG "instance": "10.142.0.107:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG [...] 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG 04:46:15 DEBUG "instance": "10.142.0.43:9100" 04:46:15 DEBUG }, 04:46:15 DEBUG "value": [ 04:46:15 DEBUG 1713584775, 04:46:15 DEBUG "0" 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG ] 04:46:15 DEBUG } 04:46:15 DEBUG } 04:46:32 INFO 04:46:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:46:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:46:32 INFO [loop_until]: OK (rc = 0) 04:46:32 DEBUG --- stdout --- 04:46:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 7m 8601Mi ds-cts-0 13m 10571Mi ds-cts-1 13m 10677Mi ds-cts-2 12m 10601Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 12m 3630Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1022m 7315Mi 04:46:32 DEBUG --- stderr --- 04:46:32 DEBUG 04:46:37 INFO 04:46:37 INFO [loop_until]: kubectl --namespace=xlou top node 04:46:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:46:37 INFO [loop_until]: OK (rc = 0) 04:46:37 DEBUG --- stdout --- 04:46:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 6275Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7357Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 11279Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 13707Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5215Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 11512Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 10788Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 6801Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 15569Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 94m 0% 7505Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 13198Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 11159Mi 19% 04:46:37 DEBUG --- stderr --- 04:46:37 DEBUG 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 ERROR [http_cmd]: ERROR 04:47:14 ERROR --- status code --- 04:47:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:47:14 ERROR --- http response --- 04:47:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:47:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:14 DEBUG "pod": "ds-idrepo-0" 04:47:14 DEBUG }, 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:14 DEBUG "pod": "ds-idrepo-2" 04:47:14 DEBUG }, 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "ds-idrepo-0" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "ds-idrepo-2" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "ds-cts-0" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "ds-cts-2" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "ds-idrepo-0" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "ds-idrepo-2" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "16359558.2434181" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584834" 04:47:14 INFO [http_cmd]: http status code OK 04:47:14 DEBUG --- status code --- 04:47:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:14 DEBUG --- http response --- 04:47:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:14 DEBUG ----- output ----- 04:47:14 DEBUG { 04:47:14 DEBUG "status": "success", 04:47:14 DEBUG "data": { 04:47:14 DEBUG "resultType": "vector", 04:47:14 DEBUG "result": [ 04:47:14 DEBUG { 04:47:14 DEBUG "metric": { 04:47:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG [...] 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG 04:47:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:14 DEBUG }, 04:47:14 DEBUG "value": [ 04:47:14 DEBUG 1713584834, 04:47:14 DEBUG "0" 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG ] 04:47:14 DEBUG } 04:47:14 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "ds-cts-2" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584835" 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "instance": "10.142.0.107:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "instance": "10.142.0.43:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "26025.6" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0.086" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "instance": "10.142.0.107:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "instance": "10.142.0.43:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "23648.199999999997" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584835" 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "ds-cts-0" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "ds-cts-2" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "instance": "10.142.0.107:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "instance": "10.142.0.43:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "3036654760960" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "ds-cts-0" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "ds-cts-2" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "ds-idrepo-0" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "ds-cts-2" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "instance": "10.142.0.107:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "instance": "10.142.0.43:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0.0005000000000469147" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "pod": "ds-cts-2" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "7767445504" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "1.0343283296061594" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584835" 04:47:15 INFO [http_cmd]: http status code OK 04:47:15 DEBUG --- status code --- 04:47:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:47:15 DEBUG --- http response --- 04:47:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:47:15 DEBUG ----- output ----- 04:47:15 DEBUG { 04:47:15 DEBUG "status": "success", 04:47:15 DEBUG "data": { 04:47:15 DEBUG "resultType": "vector", 04:47:15 DEBUG "result": [ 04:47:15 DEBUG { 04:47:15 DEBUG "metric": { 04:47:15 DEBUG "instance": "10.142.0.107:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG [...] 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG 04:47:15 DEBUG "instance": "10.142.0.43:9100" 04:47:15 DEBUG }, 04:47:15 DEBUG "value": [ 04:47:15 DEBUG 1713584835, 04:47:15 DEBUG "0" 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG ] 04:47:15 DEBUG } 04:47:15 DEBUG } 04:47:32 INFO 04:47:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:47:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:47:32 INFO [loop_until]: OK (rc = 0) 04:47:32 DEBUG --- stdout --- 04:47:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 8m 8600Mi ds-cts-0 13m 10570Mi ds-cts-1 13m 10676Mi ds-cts-2 12m 10600Mi ds-idrepo-0 13m 3721Mi ds-idrepo-1 12m 3631Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 348m 7118Mi 04:47:32 DEBUG --- stderr --- 04:47:32 DEBUG 04:47:38 INFO 04:47:38 INFO [loop_until]: kubectl --namespace=xlou top node 04:47:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:47:38 INFO [loop_until]: OK (rc = 0) 04:47:38 DEBUG --- stdout --- 04:47:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 6267Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 7353Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 11274Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3344Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 13709Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5217Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11509Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 2929Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 10777Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6806Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 15571Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 94m 0% 7500Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 70m 0% 13191Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 546m 3% 11111Mi 18% 04:47:38 DEBUG --- stderr --- 04:47:38 DEBUG 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584894" 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 ERROR [http_cmd]: ERROR 04:48:14 ERROR --- status code --- 04:48:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:48:14 ERROR --- http response --- 04:48:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:48:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "ds-idrepo-0" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "ds-idrepo-2" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "ds-idrepo-0" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "ds-idrepo-2" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "data": { 04:48:14 DEBUG "pod": "ds-cts-0" 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG }, 04:48:14 DEBUG "result": [ 04:48:14 DEBUG "value": [ 04:48:14 DEBUG { 04:48:14 DEBUG 04:48:14 DEBUG "metric": { 04:48:14 DEBUG 04:48:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:14 DEBUG 04:48:14 DEBUG }, 04:48:14 DEBUG [...] 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "ds-cts-2" 04:48:14 DEBUG [...] 04:48:14 DEBUG }, 04:48:14 DEBUG 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG 04:48:14 DEBUG "0" 04:48:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:14 DEBUG ] 04:48:14 DEBUG }, 04:48:14 DEBUG } 04:48:14 DEBUG "value": [ 04:48:14 DEBUG ] 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG } 04:48:14 DEBUG "0" 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "ds-idrepo-0" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "ds-idrepo-2" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 INFO [http_cmd]: http status code OK 04:48:14 DEBUG --- status code --- 04:48:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:14 DEBUG --- http response --- 04:48:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:48:14 DEBUG ----- output ----- 04:48:14 DEBUG { 04:48:14 DEBUG "status": "success", 04:48:14 DEBUG "data": { 04:48:14 DEBUG "resultType": "vector", 04:48:14 DEBUG "result": [ 04:48:14 DEBUG { 04:48:14 DEBUG "metric": { 04:48:14 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG [...] 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG 04:48:14 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:48:14 DEBUG }, 04:48:14 DEBUG "value": [ 04:48:14 DEBUG 1713584894, 04:48:14 DEBUG "0" 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG ] 04:48:14 DEBUG } 04:48:14 DEBUG } 04:48:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584894" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584894, 04:48:15 DEBUG "0" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "ds-cts-2" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "17296.4896797153" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "instance": "10.142.0.107:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "instance": "10.142.0.43:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "28906.066666666666" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0.086" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "instance": "10.142.0.107:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "instance": "10.142.0.43:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "26025.5" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "ds-cts-0" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "ds-cts-2" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "ds-cts-0" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "ds-cts-2" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "ds-cts-2" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "452.1958086200079" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "instance": "10.142.0.107:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "instance": "10.142.0.43:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "3036655170560" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "instance": "10.142.0.107:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "instance": "10.142.0.43:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0.00033333333334724103" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "ds-idrepo-0" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "ds-cts-2" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "pod": "ds-cts-2" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0.006295496324261745" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "7719211008" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0.7409898501040076" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584895" 04:48:15 INFO [http_cmd]: http status code OK 04:48:15 DEBUG --- status code --- 04:48:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:48:15 DEBUG --- http response --- 04:48:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:48:15 DEBUG ----- output ----- 04:48:15 DEBUG { 04:48:15 DEBUG "status": "success", 04:48:15 DEBUG "data": { 04:48:15 DEBUG "resultType": "vector", 04:48:15 DEBUG "result": [ 04:48:15 DEBUG { 04:48:15 DEBUG "metric": { 04:48:15 DEBUG "instance": "10.142.0.107:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG [...] 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG 04:48:15 DEBUG "instance": "10.142.0.43:9100" 04:48:15 DEBUG }, 04:48:15 DEBUG "value": [ 04:48:15 DEBUG 1713584895, 04:48:15 DEBUG "0.0003333333333330302" 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG ] 04:48:15 DEBUG } 04:48:15 DEBUG } 04:48:32 INFO 04:48:32 INFO [loop_until]: kubectl --namespace=xlou top pods 04:48:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:48:33 INFO [loop_until]: OK (rc = 0) 04:48:33 DEBUG --- stdout --- 04:48:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 14m 8595Mi am-767f6b67b5-qcqz2 7m 8672Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 12m 10572Mi ds-cts-1 12m 10676Mi ds-cts-2 12m 10601Mi ds-idrepo-0 15m 3722Mi ds-idrepo-1 13m 3630Mi ds-idrepo-2 12m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1003m 7315Mi 04:48:33 DEBUG --- stderr --- 04:48:33 DEBUG 04:48:38 INFO 04:48:38 INFO [loop_until]: kubectl --namespace=xlou top node 04:48:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:48:38 INFO [loop_until]: OK (rc = 0) 04:48:38 DEBUG --- stdout --- 04:48:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6277Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 7355Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 11282Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 13703Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5190Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 11513Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 66m 0% 10787Mi 9% gke-xlou-cdm-ds-0f601751-j488 72m 0% 6815Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 15554Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 7499Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 13194Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 11162Mi 19% 04:48:38 DEBUG --- stderr --- 04:48:38 DEBUG 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG [...] 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "ds-idrepo-0" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG [...] 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG "pod": "ds-idrepo-2" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713584954" 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 ERROR [http_cmd]: ERROR 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 ERROR --- status code --- 04:49:14 DEBUG --- status code --- 04:49:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 ERROR --- http response --- 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:49:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG [...] 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "ds-idrepo-0" 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG }, 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG "value": [ 04:49:14 DEBUG { 04:49:14 DEBUG 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG 04:49:14 DEBUG "data": { 04:49:14 DEBUG 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG [...] 04:49:14 DEBUG "result": [ 04:49:14 DEBUG 04:49:14 DEBUG { 04:49:14 DEBUG 04:49:14 DEBUG "metric": { 04:49:14 DEBUG 04:49:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:14 DEBUG "pod": "ds-idrepo-2" 04:49:14 DEBUG }, 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG 04:49:14 DEBUG "0" 04:49:14 DEBUG 04:49:14 DEBUG ] 04:49:14 DEBUG [...] 04:49:14 DEBUG } 04:49:14 DEBUG 04:49:14 DEBUG ] 04:49:14 DEBUG 04:49:14 DEBUG } 04:49:14 DEBUG 04:49:14 DEBUG } 04:49:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG [...] 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "ds-cts-0" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG [...] 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG "pod": "ds-cts-2" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584954" 04:49:14 INFO [http_cmd]: http status code OK 04:49:14 DEBUG --- status code --- 04:49:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:14 DEBUG --- http response --- 04:49:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:14 DEBUG ----- output ----- 04:49:14 DEBUG { 04:49:14 DEBUG "status": "success", 04:49:14 DEBUG "data": { 04:49:14 DEBUG "resultType": "vector", 04:49:14 DEBUG "result": [ 04:49:14 DEBUG { 04:49:14 DEBUG "metric": { 04:49:14 DEBUG "pod": "ds-idrepo-0" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG [...] 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG 04:49:14 DEBUG "pod": "ds-idrepo-2" 04:49:14 DEBUG }, 04:49:14 DEBUG "value": [ 04:49:14 DEBUG 1713584954, 04:49:14 DEBUG "0" 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG ] 04:49:14 DEBUG } 04:49:14 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "37010945.0801802" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "195668450.69012538" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584955" 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "instance": "10.142.0.107:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "instance": "10.142.0.43:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "28090.766666666666" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0.086" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "instance": "10.142.0.107:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "instance": "10.142.0.43:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "26589.966666666667" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "ds-cts-0" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "ds-cts-2" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "ds-cts-0" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "ds-cts-2" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "instance": "10.142.0.107:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "instance": "10.142.0.43:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "3036655522816" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "instance": "10.142.0.107:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "instance": "10.142.0.43:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0.0003000000000137485" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "ds-idrepo-0" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "ds-cts-2" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "2.0111480052080095" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "7769731072" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "1.0331054735008414" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713584955" 04:49:15 INFO [http_cmd]: http status code OK 04:49:15 DEBUG --- status code --- 04:49:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:49:15 DEBUG --- http response --- 04:49:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:49:15 DEBUG ----- output ----- 04:49:15 DEBUG { 04:49:15 DEBUG "status": "success", 04:49:15 DEBUG "data": { 04:49:15 DEBUG "resultType": "vector", 04:49:15 DEBUG "result": [ 04:49:15 DEBUG { 04:49:15 DEBUG "metric": { 04:49:15 DEBUG "instance": "10.142.0.107:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG [...] 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG 04:49:15 DEBUG "instance": "10.142.0.43:9100" 04:49:15 DEBUG }, 04:49:15 DEBUG "value": [ 04:49:15 DEBUG 1713584955, 04:49:15 DEBUG "0" 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG ] 04:49:15 DEBUG } 04:49:15 DEBUG } 04:49:33 INFO 04:49:33 INFO [loop_until]: kubectl --namespace=xlou top pods 04:49:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:49:33 INFO [loop_until]: OK (rc = 0) 04:49:33 DEBUG --- stdout --- 04:49:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8595Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 12m 10570Mi ds-cts-1 13m 10676Mi ds-cts-2 11m 10600Mi ds-idrepo-0 14m 3721Mi ds-idrepo-1 14m 3630Mi ds-idrepo-2 13m 3526Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-df4b7b554-2gxfx 6m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 997m 7315Mi 04:49:33 DEBUG --- stderr --- 04:49:33 DEBUG 04:49:38 INFO 04:49:38 INFO [loop_until]: kubectl --namespace=xlou top node 04:49:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:49:38 INFO [loop_until]: OK (rc = 0) 04:49:38 DEBUG --- stdout --- 04:49:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 79m 0% 6281Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7346Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 11272Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3359Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 13697Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5190Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 11522Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 10783Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 6815Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15547Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 7501Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 13196Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 11161Mi 19% 04:49:38 DEBUG --- stderr --- 04:49:38 DEBUG 04:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585014" 04:50:14 INFO [http_cmd]: http status code OK 04:50:14 DEBUG --- status code --- 04:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:14 DEBUG --- http response --- 04:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:14 DEBUG ----- output ----- 04:50:14 DEBUG { 04:50:14 DEBUG "status": "success", 04:50:14 DEBUG "data": { 04:50:14 DEBUG "resultType": "vector", 04:50:14 DEBUG "result": [ 04:50:14 DEBUG { 04:50:14 DEBUG "metric": { 04:50:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG [...] 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 1713585014, 04:50:14 DEBUG "0" 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG } 04:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585014" 04:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585014" 04:50:14 INFO [http_cmd]: http status code OK 04:50:14 DEBUG --- status code --- 04:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:14 DEBUG --- http response --- 04:50:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713585014" 04:50:14 INFO [http_cmd]: http status code OK 04:50:14 DEBUG --- status code --- 04:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:14 DEBUG --- http response --- 04:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:14 DEBUG ----- output ----- 04:50:14 DEBUG { 04:50:14 DEBUG "status": "success", 04:50:14 DEBUG "data": { 04:50:14 DEBUG "resultType": "vector", 04:50:14 DEBUG "result": [ 04:50:14 DEBUG { 04:50:14 DEBUG "metric": { 04:50:14 DEBUG "pod": "ds-idrepo-0" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG [...] 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG "pod": "ds-idrepo-2" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 1713585014, 04:50:14 DEBUG "0" 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG } 04:50:14 ERROR [http_cmd]: ERROR 04:50:14 ERROR --- status code --- 04:50:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:50:14 ERROR --- http response --- 04:50:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:50:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585014" 04:50:14 INFO [http_cmd]: http status code OK 04:50:14 DEBUG --- status code --- 04:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:14 DEBUG --- http response --- 04:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:14 DEBUG ----- output ----- 04:50:14 DEBUG { 04:50:14 DEBUG "status": "success", 04:50:14 DEBUG "data": { 04:50:14 DEBUG "resultType": "vector", 04:50:14 DEBUG "result": [ 04:50:14 DEBUG { 04:50:14 DEBUG "metric": { 04:50:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG [...] 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 1713585014, 04:50:14 DEBUG "0" 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG } 04:50:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585014" 04:50:14 INFO [http_cmd]: http status code OK 04:50:14 DEBUG --- status code --- 04:50:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:14 DEBUG --- http response --- 04:50:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:14 DEBUG ----- output ----- 04:50:14 DEBUG { 04:50:14 DEBUG "status": "success", 04:50:14 DEBUG "data": { 04:50:14 DEBUG "resultType": "vector", 04:50:14 DEBUG "result": [ 04:50:14 DEBUG { 04:50:14 DEBUG "metric": { 04:50:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG [...] 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG 04:50:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:14 DEBUG }, 04:50:14 DEBUG "value": [ 04:50:14 DEBUG 1713585014, 04:50:14 DEBUG "0" 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG ] 04:50:14 DEBUG } 04:50:14 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "ds-cts-0" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "ds-idrepo-0" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-idrepo-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "ds-idrepo-0" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-idrepo-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "388175847.47005624" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "instance": "10.142.0.107:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "instance": "10.142.0.43:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "26940.766666666666" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0.086" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "instance": "10.142.0.107:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "instance": "10.142.0.43:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "24944.8" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "ds-cts-0" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "715.1587107588085" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "ds-cts-0" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "instance": "10.142.0.107:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "instance": "10.142.0.43:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "3036656149504" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "instance": "10.142.0.107:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "instance": "10.142.0.43:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0.0003666666666807335" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "ds-idrepo-0" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "pod": "ds-cts-2" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0.01073233927982703" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "7770185728" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "1.035018084905706" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585015" 04:50:15 INFO [http_cmd]: http status code OK 04:50:15 DEBUG --- status code --- 04:50:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:50:15 DEBUG --- http response --- 04:50:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:50:15 DEBUG ----- output ----- 04:50:15 DEBUG { 04:50:15 DEBUG "status": "success", 04:50:15 DEBUG "data": { 04:50:15 DEBUG "resultType": "vector", 04:50:15 DEBUG "result": [ 04:50:15 DEBUG { 04:50:15 DEBUG "metric": { 04:50:15 DEBUG "instance": "10.142.0.107:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG [...] 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG 04:50:15 DEBUG "instance": "10.142.0.43:9100" 04:50:15 DEBUG }, 04:50:15 DEBUG "value": [ 04:50:15 DEBUG 1713585015, 04:50:15 DEBUG "0" 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG ] 04:50:15 DEBUG } 04:50:15 DEBUG } 04:50:33 INFO 04:50:33 INFO [loop_until]: kubectl --namespace=xlou top pods 04:50:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:50:33 INFO [loop_until]: OK (rc = 0) 04:50:33 DEBUG --- stdout --- 04:50:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8595Mi am-767f6b67b5-qcqz2 13m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 12m 10570Mi ds-cts-1 14m 10677Mi ds-cts-2 11m 10601Mi ds-idrepo-0 14m 3720Mi ds-idrepo-1 13m 3631Mi ds-idrepo-2 13m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-df4b7b554-2gxfx 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1010m 7318Mi 04:50:33 DEBUG --- stderr --- 04:50:33 DEBUG 04:50:38 INFO 04:50:38 INFO [loop_until]: kubectl --namespace=xlou top node 04:50:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:50:38 INFO [loop_until]: OK (rc = 0) 04:50:38 DEBUG --- stdout --- 04:50:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6285Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 7345Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 11266Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3369Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 13695Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11523Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 2940Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 10775Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6804Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 15550Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 97m 0% 7492Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13163Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1046m 6% 11159Mi 19% 04:50:38 DEBUG --- stderr --- 04:50:38 DEBUG 04:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585074" 04:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713585074" 04:51:14 INFO [http_cmd]: http status code OK 04:51:14 DEBUG --- status code --- 04:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:14 DEBUG --- http response --- 04:51:14 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:14 DEBUG ----- output ----- 04:51:14 DEBUG { 04:51:14 DEBUG "status": "success", 04:51:14 DEBUG "data": { 04:51:14 DEBUG "resultType": "vector", 04:51:14 DEBUG "result": [ 04:51:14 DEBUG { 04:51:14 DEBUG "metric": { 04:51:14 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:14 DEBUG }, 04:51:14 DEBUG "value": [ 04:51:14 DEBUG 04:51:14 DEBUG 04:51:14 DEBUG 04:51:14 DEBUG [...] 04:51:14 DEBUG 04:51:14 DEBUG 04:51:14 DEBUG 04:51:14 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:14 DEBUG }, 04:51:14 DEBUG "value": [ 04:51:14 DEBUG 1713585074, 04:51:14 DEBUG "0" 04:51:14 DEBUG ] 04:51:14 DEBUG } 04:51:14 DEBUG ] 04:51:14 DEBUG } 04:51:14 DEBUG } 04:51:14 ERROR [http_cmd]: ERROR 04:51:14 ERROR --- status code --- 04:51:14 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:51:14 ERROR --- http response --- 04:51:14 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:51:14 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585074" 04:51:14 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585074" 04:51:14 INFO [http_cmd]: http status code OK 04:51:14 DEBUG --- status code --- 04:51:14 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:14 DEBUG --- http response --- 04:51:14 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-idrepo-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-idrepo-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585074, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-cts-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-cts-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-idrepo-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-idrepo-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-idrepo-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-idrepo-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "747.607929112999" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "instance": "10.142.0.107:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "instance": "10.142.0.43:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "27669.166666666668" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "am-767f6b67b5-9zw22" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "am-767f6b67b5-vbltz" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0.0865" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "instance": "10.142.0.107:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "instance": "10.142.0.43:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "24826.566666666666" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-cts-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-cts-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "instance": "10.142.0.107:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "instance": "10.142.0.43:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "3036656475136" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-cts-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-cts-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "instance": "10.142.0.107:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "instance": "10.142.0.43:9100" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0.0003333333333462936" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "ds-idrepo-0" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "ds-cts-2" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "pod": "idm-b85f9d978-wnhxb" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "0.0012657863741205048" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713585075" 04:51:15 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713585075" 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "7772917760" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:15 INFO [http_cmd]: http status code OK 04:51:15 DEBUG --- status code --- 04:51:15 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:15 DEBUG --- http response --- 04:51:15 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:15 DEBUG ----- output ----- 04:51:15 DEBUG { 04:51:15 DEBUG "status": "success", 04:51:15 DEBUG "data": { 04:51:15 DEBUG "resultType": "vector", 04:51:15 DEBUG "result": [ 04:51:15 DEBUG { 04:51:15 DEBUG "metric": { 04:51:15 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG [...] 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG 04:51:15 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:51:15 DEBUG }, 04:51:15 DEBUG "value": [ 04:51:15 DEBUG 1713585075, 04:51:15 DEBUG "1.0393405603475867" 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG ] 04:51:15 DEBUG } 04:51:15 DEBUG } 04:51:16 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713585076" 04:51:16 INFO [http_cmd]: http status code OK 04:51:16 DEBUG --- status code --- 04:51:16 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:51:16 DEBUG --- http response --- 04:51:16 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:51:16 DEBUG ----- output ----- 04:51:16 DEBUG { 04:51:16 DEBUG "status": "success", 04:51:16 DEBUG "data": { 04:51:16 DEBUG "resultType": "vector", 04:51:16 DEBUG "result": [ 04:51:16 DEBUG { 04:51:16 DEBUG "metric": { 04:51:16 DEBUG "instance": "10.142.0.107:9100" 04:51:16 DEBUG }, 04:51:16 DEBUG "value": [ 04:51:16 DEBUG 04:51:16 DEBUG 04:51:16 DEBUG 04:51:16 DEBUG [...] 04:51:16 DEBUG 04:51:16 DEBUG 04:51:16 DEBUG 04:51:16 DEBUG "instance": "10.142.0.43:9100" 04:51:16 DEBUG }, 04:51:16 DEBUG "value": [ 04:51:16 DEBUG 1713585076, 04:51:16 DEBUG "0" 04:51:16 DEBUG ] 04:51:16 DEBUG } 04:51:16 DEBUG ] 04:51:16 DEBUG } 04:51:16 DEBUG } 04:51:24 INFO Finished: True 04:51:24 INFO Waiting for threads to register finish flag 04:52:16 INFO Done. Have a nice day! :) 127.0.0.1 - - [20/Apr/2024 04:52:16] "GET /monitoring/stop HTTP/1.1" 200 - 04:52:20 WARNING File /tmp/lodemon_data-20240419_220641/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6481', ...) 127.0.0.1 - - [20/Apr/2024 04:52:25] "GET /monitoring/process HTTP/1.1" 200 -