==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-5d8fc565dd-9g64c Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Sun, 14 Apr 2024 17:27:28 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5d8fc565dd skaffold.dev/run-id=3616b385-22d9-46d2-b245-fb1112263b63 Annotations: Status: Running IP: 10.106.46.140 IPs: IP: 10.106.46.140 Controlled By: ReplicaSet/lodemon-5d8fc565dd Containers: lodemon: Container ID: containerd://742330277a3c1762c17b2fd55de1041e772fbb8fe097b6974d93ec4494c31780 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 14 Apr 2024 17:27:29 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7h5cd (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-7h5cd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 00:33:20 INFO 00:33:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:33:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:33:20 INFO [loop_until]: OK (rc = 0) 00:33:20 DEBUG --- stdout --- 00:33:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12687m 8614Mi am-7d597cf7d6-8vr8f 11607m 8933Mi am-7d597cf7d6-9wz9h 15930m 8819Mi ds-cts-0 11525m 8139Mi ds-cts-1 12010m 8487Mi ds-cts-2 12139m 8181Mi ds-idrepo-0 13325m 4577Mi ds-idrepo-1 2046m 4303Mi ds-idrepo-2 1877m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6489m 5568Mi 00:33:20 DEBUG --- stderr --- 00:33:20 DEBUG 00:33:24 INFO 00:33:24 INFO [loop_until]: kubectl --namespace=xlou top node 00:33:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:33:24 INFO [loop_until]: OK (rc = 0) 00:33:24 DEBUG --- stdout --- 00:33:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 4863Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12053m 40% 11057Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13139m 44% 10694Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17513m 58% 3054Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2227m 7% 6822Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11744m 39% 12305Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12470m 41% 10402Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1979m 6% 5970Mi 5% gke-xlou-cdm-ds-0f601751-j488 12202m 40% 10708Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2242Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16948m 56% 10421Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13361m 44% 6488Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6316m 39% 9309Mi 15% 00:33:24 DEBUG --- stderr --- 00:33:24 DEBUG 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713137643" 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137643" 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 ERROR [http_cmd]: ERROR 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 ERROR --- status code --- 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG --- status code --- 00:34:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 ERROR --- http response --- 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "instance": "10.142.0.107:9100" 00:34:03 DEBUG "pod": "ds-idrepo-1" 00:34:03 DEBUG }, 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "instance": "10.142.0.43:9100" 00:34:03 DEBUG "pod": "ds-idrepo-0" 00:34:03 DEBUG }, 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0.00046666666666700016" 00:34:03 DEBUG "0" 00:34:03 DEBUG ] 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "2289.9666666666667" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "instance": "10.142.0.107:9100" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "instance": "10.142.0.43:9100" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "13373688832" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "ds-idrepo-1" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "ds-idrepo-0" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "instance": "10.142.0.107:9100" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "instance": "10.142.0.43:9100" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "14724.966666666665" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "ds-idrepo-1" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "ds-cts-1" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "5934841856" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "ds-idrepo-1" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "ds-idrepo-0" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-8vr8f" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "412821.5" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "6.416773888398813" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "2290.1666666666665" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "0.051" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:03 INFO [http_cmd]: http status code OK 00:34:03 DEBUG --- status code --- 00:34:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:03 DEBUG --- http response --- 00:34:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:34:03 DEBUG ----- output ----- 00:34:03 DEBUG { 00:34:03 DEBUG "status": "success", 00:34:03 DEBUG "data": { 00:34:03 DEBUG "resultType": "vector", 00:34:03 DEBUG "result": [ 00:34:03 DEBUG { 00:34:03 DEBUG "metric": { 00:34:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG [...] 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG 00:34:03 DEBUG "pod": "am-7d597cf7d6-8vr8f" 00:34:03 DEBUG }, 00:34:03 DEBUG "value": [ 00:34:03 DEBUG 1713137643, 00:34:03 DEBUG "334287398.99404275" 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG ] 00:34:03 DEBUG } 00:34:03 DEBUG } 00:34:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137643" 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137644" 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137644" 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137644" 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "ds-idrepo-1" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "ds-cts-1" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137643, 00:34:04 DEBUG "0.0002" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "2292.0666666666666" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "ds-idrepo-1" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "ds-idrepo-0" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "0" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "am-7d597cf7d6-8vr8f" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "8319410.979916528" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137644" 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "instance": "10.142.0.107:9100" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "instance": "10.142.0.43:9100" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "0.00033333333333333365" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137644" 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "ds-idrepo-1" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "ds-idrepo-0" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "0" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137644" 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "am-7d597cf7d6-8vr8f" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "11.538333785181718" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137644" 00:34:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137644" 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "2290.1" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:04 INFO [http_cmd]: http status code OK 00:34:04 DEBUG --- status code --- 00:34:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:34:04 DEBUG --- http response --- 00:34:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:34:04 DEBUG ----- output ----- 00:34:04 DEBUG { 00:34:04 DEBUG "status": "success", 00:34:04 DEBUG "data": { 00:34:04 DEBUG "resultType": "vector", 00:34:04 DEBUG "result": [ 00:34:04 DEBUG { 00:34:04 DEBUG "metric": { 00:34:04 DEBUG "instance": "10.142.0.107:9100" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG [...] 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG 00:34:04 DEBUG "instance": "10.142.0.43:9100" 00:34:04 DEBUG }, 00:34:04 DEBUG "value": [ 00:34:04 DEBUG 1713137644, 00:34:04 DEBUG "5485.133333333333" 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG ] 00:34:04 DEBUG } 00:34:04 DEBUG } 00:34:20 INFO 00:34:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:34:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:34:20 INFO [loop_until]: OK (rc = 0) 00:34:20 DEBUG --- stdout --- 00:34:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12676m 8611Mi am-7d597cf7d6-8vr8f 11419m 8935Mi am-7d597cf7d6-9wz9h 15961m 8825Mi ds-cts-0 11701m 8120Mi ds-cts-1 11953m 8513Mi ds-cts-2 12254m 8206Mi ds-idrepo-0 13227m 4576Mi ds-idrepo-1 1959m 4303Mi ds-idrepo-2 1864m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6420m 5569Mi 00:34:20 DEBUG --- stderr --- 00:34:20 DEBUG 00:34:24 INFO 00:34:24 INFO [loop_until]: kubectl --namespace=xlou top node 00:34:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:34:24 INFO [loop_until]: OK (rc = 0) 00:34:24 DEBUG --- stdout --- 00:34:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4849Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12010m 40% 11063Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12979m 43% 10684Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17491m 58% 3045Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2188m 7% 6830Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11621m 38% 12261Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11841m 39% 10408Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2013m 6% 5976Mi 5% gke-xlou-cdm-ds-0f601751-j488 12041m 40% 10715Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2249Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17018m 57% 10416Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13357m 44% 6494Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6537m 41% 9309Mi 15% 00:34:24 DEBUG --- stderr --- 00:34:24 DEBUG 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137703" 00:35:03 ERROR [http_cmd]: ERROR 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 ERROR --- status code --- 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 ERROR --- http response --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG { 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "data": { 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG "result": [ 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG { 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "ds-idrepo-1" 00:35:03 DEBUG "instance": "10.142.0.107:9100" 00:35:03 DEBUG "instance": "10.142.0.107:9100" 00:35:03 DEBUG }, 00:35:03 DEBUG }, 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG "value": [ 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG [...] 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "ds-idrepo-0" 00:35:03 DEBUG "instance": "10.142.0.43:9100" 00:35:03 DEBUG "instance": "10.142.0.43:9100" 00:35:03 DEBUG }, 00:35:03 DEBUG }, 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG "value": [ 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "0" 00:35:03 DEBUG "0.00036666666666652263" 00:35:03 DEBUG "13375151104" 00:35:03 DEBUG ] 00:35:03 DEBUG ] 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG ] 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713137703" 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "2285.7" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137703" 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "5937549312" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "ds-idrepo-1" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "ds-idrepo-0" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "0" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "instance": "10.142.0.107:9100" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "instance": "10.142.0.43:9100" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "14727.533333333331" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "412714" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "ds-idrepo-1" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "ds-cts-1" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "0" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "ds-idrepo-1" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "ds-idrepo-0" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "0" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "ds-cts-1" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "30252273.29643408" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137703" 00:35:03 INFO [http_cmd]: http status code OK 00:35:03 DEBUG --- status code --- 00:35:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:03 DEBUG --- http response --- 00:35:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:03 DEBUG ----- output ----- 00:35:03 DEBUG { 00:35:03 DEBUG "status": "success", 00:35:03 DEBUG "data": { 00:35:03 DEBUG "resultType": "vector", 00:35:03 DEBUG "result": [ 00:35:03 DEBUG { 00:35:03 DEBUG "metric": { 00:35:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG [...] 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG 00:35:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:03 DEBUG }, 00:35:03 DEBUG "value": [ 00:35:03 DEBUG 1713137703, 00:35:03 DEBUG "0" 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG ] 00:35:03 DEBUG } 00:35:03 DEBUG } 00:35:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713137703" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137703, 00:35:04 DEBUG "6.380472545454085" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "2286.5333333333333" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "0.051" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "ds-cts-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "42323227.29334906" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "ds-cts-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "269368336.5695793" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "2296.7666666666664" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "ds-idrepo-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "ds-cts-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "0.00006666666666666667" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "ds-idrepo-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "ds-idrepo-0" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "0" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "instance": "10.142.0.107:9100" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "instance": "10.142.0.43:9100" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "0.000333333333333341" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "ds-idrepo-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "ds-idrepo-0" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "0" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "ds-cts-1" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "11.90275201887289" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "instance": "10.142.0.107:9100" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "instance": "10.142.0.43:9100" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "6951.333333333333" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137704" 00:35:04 INFO [http_cmd]: http status code OK 00:35:04 DEBUG --- status code --- 00:35:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:35:04 DEBUG --- http response --- 00:35:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:35:04 DEBUG ----- output ----- 00:35:04 DEBUG { 00:35:04 DEBUG "status": "success", 00:35:04 DEBUG "data": { 00:35:04 DEBUG "resultType": "vector", 00:35:04 DEBUG "result": [ 00:35:04 DEBUG { 00:35:04 DEBUG "metric": { 00:35:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG [...] 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG 00:35:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:35:04 DEBUG }, 00:35:04 DEBUG "value": [ 00:35:04 DEBUG 1713137704, 00:35:04 DEBUG "2288.766666666667" 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG ] 00:35:04 DEBUG } 00:35:04 DEBUG } 00:35:20 INFO 00:35:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:35:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:35:20 INFO [loop_until]: OK (rc = 0) 00:35:20 DEBUG --- stdout --- 00:35:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12189m 8678Mi am-7d597cf7d6-8vr8f 11602m 8935Mi am-7d597cf7d6-9wz9h 14940m 8820Mi ds-cts-0 11545m 8115Mi ds-cts-1 11716m 8500Mi ds-cts-2 11584m 8181Mi ds-idrepo-0 13143m 4577Mi ds-idrepo-1 2044m 4303Mi ds-idrepo-2 1887m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1014Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6397m 5568Mi 00:35:20 DEBUG --- stderr --- 00:35:20 DEBUG 00:35:24 INFO 00:35:24 INFO [loop_until]: kubectl --namespace=xlou top node 00:35:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:35:25 INFO [loop_until]: OK (rc = 0) 00:35:25 DEBUG --- stdout --- 00:35:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 4871Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11848m 39% 11069Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13043m 43% 10745Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16981m 56% 3043Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2220m 7% 6836Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11315m 37% 12292Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2924Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12335m 41% 10402Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1992m 6% 5976Mi 5% gke-xlou-cdm-ds-0f601751-j488 11894m 39% 10717Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2238Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16804m 56% 10415Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13345m 44% 6492Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6543m 41% 9311Mi 15% 00:35:25 DEBUG --- stderr --- 00:35:25 DEBUG 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713137763" 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137763" 00:36:03 ERROR [http_cmd]: ERROR 00:36:03 ERROR --- status code --- 00:36:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:36:03 ERROR --- http response --- 00:36:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:36:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137763" 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "pod": "ds-idrepo-1" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "pod": "ds-idrepo-0" 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:03 DEBUG }, 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG "value": [ 00:36:03 DEBUG { 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "0" 00:36:03 DEBUG "data": { 00:36:03 DEBUG ] 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG } 00:36:03 DEBUG "result": [ 00:36:03 DEBUG ] 00:36:03 DEBUG { 00:36:03 DEBUG } 00:36:03 DEBUG "metric": { 00:36:03 DEBUG } 00:36:03 DEBUG "instance": "10.142.0.107:9100" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "instance": "10.142.0.43:9100" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "0.00043333333333350765" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "instance": "10.142.0.107:9100" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "instance": "10.142.0.43:9100" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "13375726592" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "pod": "ds-idrepo-1" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:03 DEBUG [...] 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG 00:36:03 DEBUG { 00:36:03 DEBUG 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG 00:36:03 DEBUG "data": { 00:36:03 DEBUG "pod": "ds-idrepo-0" 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG }, 00:36:03 DEBUG "result": [ 00:36:03 DEBUG "value": [ 00:36:03 DEBUG { 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "0" 00:36:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:03 DEBUG ] 00:36:03 DEBUG }, 00:36:03 DEBUG } 00:36:03 DEBUG "value": [ 00:36:03 DEBUG ] 00:36:03 DEBUG 00:36:03 DEBUG } 00:36:03 DEBUG 00:36:03 DEBUG } 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "2320.2666666666664" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137763" 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "instance": "10.142.0.107:9100" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "instance": "10.142.0.43:9100" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "14842.133333333331" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713137763" 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "412800.5" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "5941448704" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "pod": "ds-idrepo-1" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "pod": "ds-idrepo-0" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "0" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "0" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713137763" 00:36:03 INFO [http_cmd]: http status code OK 00:36:03 DEBUG --- status code --- 00:36:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:03 DEBUG --- http response --- 00:36:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:36:03 DEBUG ----- output ----- 00:36:03 DEBUG { 00:36:03 DEBUG "status": "success", 00:36:03 DEBUG "data": { 00:36:03 DEBUG "resultType": "vector", 00:36:03 DEBUG "result": [ 00:36:03 DEBUG { 00:36:03 DEBUG "metric": { 00:36:03 DEBUG "pod": "ds-idrepo-1" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG [...] 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG 00:36:03 DEBUG "pod": "ds-cts-1" 00:36:03 DEBUG }, 00:36:03 DEBUG "value": [ 00:36:03 DEBUG 1713137763, 00:36:03 DEBUG "0" 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG ] 00:36:03 DEBUG } 00:36:03 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "2319.9" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0.051" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "6.357382901043068" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "469.585933674605" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "2290.2666666666664" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "30656213.301046073" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137764" 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "ds-idrepo-1" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "ds-idrepo-0" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:36:04 DEBUG } 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "ds-idrepo-1" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "ds-cts-1" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0.0005113333333333333" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "instance": "10.142.0.107:9100" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "instance": "10.142.0.43:9100" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0.001694109214332852" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "ds-idrepo-1" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "ds-idrepo-0" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "0" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "instance": "10.142.0.107:9100" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "instance": "10.142.0.43:9100" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "5583.166666666667" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137764" 00:36:04 INFO [http_cmd]: http status code OK 00:36:04 DEBUG --- status code --- 00:36:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:36:04 DEBUG --- http response --- 00:36:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:36:04 DEBUG ----- output ----- 00:36:04 DEBUG { 00:36:04 DEBUG "status": "success", 00:36:04 DEBUG "data": { 00:36:04 DEBUG "resultType": "vector", 00:36:04 DEBUG "result": [ 00:36:04 DEBUG { 00:36:04 DEBUG "metric": { 00:36:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG [...] 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG 00:36:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:36:04 DEBUG }, 00:36:04 DEBUG "value": [ 00:36:04 DEBUG 1713137764, 00:36:04 DEBUG "2313.3333333333335" 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG ] 00:36:04 DEBUG } 00:36:04 DEBUG } 00:36:20 INFO 00:36:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:36:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:36:20 INFO [loop_until]: OK (rc = 0) 00:36:20 DEBUG --- stdout --- 00:36:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12523m 8615Mi am-7d597cf7d6-8vr8f 11616m 8936Mi am-7d597cf7d6-9wz9h 15476m 8821Mi ds-cts-0 11513m 8141Mi ds-cts-1 11799m 8497Mi ds-cts-2 12015m 8184Mi ds-idrepo-0 13143m 4577Mi ds-idrepo-1 1981m 4303Mi ds-idrepo-2 1875m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1014Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6408m 5573Mi 00:36:20 DEBUG --- stderr --- 00:36:20 DEBUG 00:36:25 INFO 00:36:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:36:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:36:25 INFO [loop_until]: OK (rc = 0) 00:36:25 DEBUG --- stdout --- 00:36:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4883Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12186m 40% 11067Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13096m 43% 10701Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17640m 59% 3046Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2250m 7% 6828Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11611m 38% 12293Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2925Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12368m 41% 10410Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1988m 6% 5970Mi 5% gke-xlou-cdm-ds-0f601751-j488 12003m 40% 10716Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2250Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17033m 57% 10420Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13377m 44% 6491Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6558m 41% 9312Mi 15% 00:36:25 DEBUG --- stderr --- 00:36:25 DEBUG 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137823" 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137823" 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137823" 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713137823" 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:37:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:03 DEBUG ----- output ----- 00:37:03 DEBUG ----- output ----- 00:37:03 DEBUG { 00:37:03 DEBUG { 00:37:03 DEBUG "status": "success", 00:37:03 DEBUG "status": "success", 00:37:03 DEBUG "data": { 00:37:03 DEBUG "data": { 00:37:03 DEBUG "resultType": "vector", 00:37:03 DEBUG "resultType": "vector", 00:37:03 DEBUG "result": [ 00:37:03 DEBUG "result": [ 00:37:03 DEBUG { 00:37:03 DEBUG { 00:37:03 DEBUG "metric": { 00:37:03 DEBUG "metric": { 00:37:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:03 DEBUG "pod": "ds-idrepo-1" 00:37:03 DEBUG }, 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG [...] 00:37:03 DEBUG [...] 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:03 DEBUG "pod": "ds-idrepo-0" 00:37:03 DEBUG }, 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 1713137823, 00:37:03 DEBUG 1713137823, 00:37:03 DEBUG "2301.1666666666665" 00:37:03 DEBUG "0" 00:37:03 DEBUG ] 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 DEBUG ] 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137823" 00:37:03 ERROR [http_cmd]: ERROR 00:37:03 ERROR --- status code --- 00:37:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:37:03 ERROR --- http response --- 00:37:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:37:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137823" 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137823" 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:03 DEBUG ----- output ----- 00:37:03 DEBUG { 00:37:03 DEBUG "status": "success", 00:37:03 DEBUG "data": { 00:37:03 DEBUG "resultType": "vector", 00:37:03 DEBUG "result": [ 00:37:03 DEBUG { 00:37:03 DEBUG "metric": { 00:37:03 DEBUG "instance": "10.142.0.107:9100" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG [...] 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG "instance": "10.142.0.43:9100" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 1713137823, 00:37:03 DEBUG "13376365568" 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:03 DEBUG ----- output ----- 00:37:03 DEBUG { 00:37:03 DEBUG "status": "success", 00:37:03 DEBUG "data": { 00:37:03 DEBUG "resultType": "vector", 00:37:03 DEBUG "result": [ 00:37:03 DEBUG { 00:37:03 DEBUG "metric": { 00:37:03 DEBUG "instance": "10.142.0.107:9100" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG [...] 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG "instance": "10.142.0.43:9100" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 1713137823, 00:37:03 DEBUG "0.00043333333333303397" 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:37:03 DEBUG ----- output ----- 00:37:03 DEBUG { 00:37:03 DEBUG "status": "success", 00:37:03 DEBUG "data": { 00:37:03 DEBUG "resultType": "vector", 00:37:03 DEBUG "result": [ 00:37:03 DEBUG { 00:37:03 DEBUG "metric": { 00:37:03 DEBUG "pod": "ds-idrepo-1" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG [...] 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG "pod": "ds-idrepo-0" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 1713137823, 00:37:03 DEBUG "0" 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG ] 00:37:03 DEBUG } 00:37:03 DEBUG } 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137823" 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137823" 00:37:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137823" 00:37:03 INFO [http_cmd]: http status code OK 00:37:03 DEBUG --- status code --- 00:37:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:03 DEBUG --- http response --- 00:37:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 00:37:03 DEBUG ----- output ----- 00:37:03 DEBUG { 00:37:03 DEBUG "status": "success", 00:37:03 DEBUG "data": { 00:37:03 DEBUG "resultType": "vector", 00:37:03 DEBUG "result": [ 00:37:03 DEBUG { 00:37:03 DEBUG "metric": { 00:37:03 DEBUG "pod": "ds-cts-2" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG [...] 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG 00:37:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:37:03 DEBUG }, 00:37:03 DEBUG "value": [ 00:37:03 DEBUG 1713137823, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "instance": "10.142.0.107:9100" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "instance": "10.142.0.43:9100" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137823, 00:37:04 DEBUG "14606.733333333332" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137823, 00:37:04 DEBUG "413126.5" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "5941342208" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-idrepo-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "ds-idrepo-0" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-idrepo-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "ds-cts-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0.051" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "2300.6666666666665" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713137824" 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-cts-2" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "6.403151729522688" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "2289.6" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137824" 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "instance": "10.142.0.107:9100" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "instance": "10.142.0.43:9100" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-cts-2" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-idrepo-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "ds-cts-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0.00019308333333333333" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-idrepo-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "ds-idrepo-0" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-cts-2" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "ds-idrepo-1" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "ds-idrepo-0" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "0" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "instance": "10.142.0.107:9100" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "instance": "10.142.0.43:9100" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "5383.333333333334" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137824" 00:37:04 INFO [http_cmd]: http status code OK 00:37:04 DEBUG --- status code --- 00:37:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:37:04 DEBUG --- http response --- 00:37:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:37:04 DEBUG ----- output ----- 00:37:04 DEBUG { 00:37:04 DEBUG "status": "success", 00:37:04 DEBUG "data": { 00:37:04 DEBUG "resultType": "vector", 00:37:04 DEBUG "result": [ 00:37:04 DEBUG { 00:37:04 DEBUG "metric": { 00:37:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG [...] 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG 00:37:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:37:04 DEBUG }, 00:37:04 DEBUG "value": [ 00:37:04 DEBUG 1713137824, 00:37:04 DEBUG "2299.7" 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG ] 00:37:04 DEBUG } 00:37:04 DEBUG } 00:37:20 INFO 00:37:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:37:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:37:20 INFO [loop_until]: OK (rc = 0) 00:37:20 DEBUG --- stdout --- 00:37:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12323m 8614Mi am-7d597cf7d6-8vr8f 11488m 8935Mi am-7d597cf7d6-9wz9h 15985m 8833Mi ds-cts-0 11694m 8127Mi ds-cts-1 11646m 8481Mi ds-cts-2 12393m 8185Mi ds-idrepo-0 13397m 4577Mi ds-idrepo-1 2032m 4303Mi ds-idrepo-2 1817m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1014Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 3m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6497m 5572Mi 00:37:20 DEBUG --- stderr --- 00:37:20 DEBUG 00:37:25 INFO 00:37:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:37:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:37:25 INFO [loop_until]: OK (rc = 0) 00:37:25 DEBUG --- stdout --- 00:37:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4883Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12141m 40% 11068Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12984m 43% 10691Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17546m 58% 3044Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2240m 7% 6840Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11574m 38% 12285Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12327m 41% 10413Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2022m 6% 5976Mi 5% gke-xlou-cdm-ds-0f601751-j488 12035m 40% 10710Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2245Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17099m 57% 10474Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13467m 45% 6497Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6553m 41% 9314Mi 15% 00:37:25 DEBUG --- stderr --- 00:37:25 DEBUG 00:38:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137883" 00:38:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713137883" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 ERROR [http_cmd]: ERROR 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 ERROR --- status code --- 00:38:04 DEBUG --- status code --- 00:38:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 ERROR --- http response --- 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:38:04 DEBUG ----- output ----- 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG { 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG "data": { 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG "result": [ 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG }, 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "value": [ 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG [...] 00:38:04 DEBUG "instance": "10.142.0.107:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG }, 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG "value": [ 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 1713137883, 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "2285.0333333333333" 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG ] 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-idrepo-0" 00:38:04 DEBUG } 00:38:04 DEBUG "instance": "10.142.0.43:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG ] 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG } 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG } 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG "13377070080" 00:38:04 DEBUG ] 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG { 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG "data": { 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG "result": [ 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG { 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG }, 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG "value": [ 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG 00:38:04 DEBUG "instance": "10.142.0.107:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG [...] 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG }, 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "413325.5" 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-idrepo-0" 00:38:04 DEBUG ] 00:38:04 DEBUG "instance": "10.142.0.43:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG } 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG ] 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG } 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG } 00:38:04 DEBUG "0.0003666666666669964" 00:38:04 DEBUG ] 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "instance": "10.142.0.107:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "instance": "10.142.0.43:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "14374.699999999999" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-cts-2" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-idrepo-0" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "5942272000" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-cts-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "2285.1" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0.052" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG { 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG "value": [ 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG { 00:38:04 DEBUG "2285.7666666666664" 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG ] 00:38:04 DEBUG "data": { 00:38:04 DEBUG } 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG ] 00:38:04 DEBUG "result": [ 00:38:04 DEBUG } 00:38:04 DEBUG { 00:38:04 DEBUG } 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-cts-2" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "6.505874277538545" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "instance": "10.142.0.107:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "instance": "10.142.0.43:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0.00033333333333333365" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-cts-2" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-cts-2" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-cts-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0.000033333333333333335" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-idrepo-0" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "ds-idrepo-1" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "ds-idrepo-0" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "0" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "instance": "10.142.0.107:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "instance": "10.142.0.43:9100" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "5488.066666666667" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137884" 00:38:04 INFO [http_cmd]: http status code OK 00:38:04 DEBUG --- status code --- 00:38:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:38:04 DEBUG --- http response --- 00:38:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:38:04 DEBUG ----- output ----- 00:38:04 DEBUG { 00:38:04 DEBUG "status": "success", 00:38:04 DEBUG "data": { 00:38:04 DEBUG "resultType": "vector", 00:38:04 DEBUG "result": [ 00:38:04 DEBUG { 00:38:04 DEBUG "metric": { 00:38:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG [...] 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG 00:38:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:38:04 DEBUG }, 00:38:04 DEBUG "value": [ 00:38:04 DEBUG 1713137884, 00:38:04 DEBUG "2285.0666666666666" 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG ] 00:38:04 DEBUG } 00:38:04 DEBUG } 00:38:20 INFO 00:38:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:38:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:38:20 INFO [loop_until]: OK (rc = 0) 00:38:20 DEBUG --- stdout --- 00:38:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12550m 8617Mi am-7d597cf7d6-8vr8f 11690m 8932Mi am-7d597cf7d6-9wz9h 15852m 8819Mi ds-cts-0 11423m 8142Mi ds-cts-1 12093m 8469Mi ds-cts-2 12128m 8185Mi ds-idrepo-0 13335m 4577Mi ds-idrepo-1 1992m 4304Mi ds-idrepo-2 1905m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1014Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6341m 5572Mi 00:38:20 DEBUG --- stderr --- 00:38:20 DEBUG 00:38:25 INFO 00:38:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:38:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:38:25 INFO [loop_until]: OK (rc = 0) 00:38:25 DEBUG --- stdout --- 00:38:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4886Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12083m 40% 11074Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13055m 43% 10688Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17519m 58% 3040Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2201m 7% 6822Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11755m 39% 12272Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2927Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12449m 41% 10409Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2010m 6% 5984Mi 5% gke-xlou-cdm-ds-0f601751-j488 12028m 40% 10715Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2243Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16962m 56% 10422Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13402m 44% 6498Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6547m 41% 9316Mi 15% 00:38:25 DEBUG --- stderr --- 00:38:25 DEBUG 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 ERROR [http_cmd]: ERROR 00:39:04 DEBUG --- status code --- 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 ERROR --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- status code --- 00:39:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 ERROR --- http response --- 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:39:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:39:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG "metric": { 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG }, 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG }, 00:39:04 DEBUG "instance": "10.142.0.107:9100" 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG }, 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG "value": [ 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG { 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "data": { 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "pod": "ds-idrepo-0" 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG "result": [ 00:39:04 DEBUG }, 00:39:04 DEBUG "pod": "ds-idrepo-0" 00:39:04 DEBUG }, 00:39:04 DEBUG "instance": "10.142.0.43:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG { 00:39:04 DEBUG "value": [ 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG "metric": { 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "instance": "10.142.0.107:9100" 00:39:04 DEBUG "0" 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "412920" 00:39:04 DEBUG }, 00:39:04 DEBUG ] 00:39:04 DEBUG "2298" 00:39:04 DEBUG ] 00:39:04 DEBUG "13377778688" 00:39:04 DEBUG ] 00:39:04 DEBUG "value": [ 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG [...] 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG 00:39:04 DEBUG { 00:39:04 DEBUG 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "instance": "10.142.0.43:9100" 00:39:04 DEBUG "data": { 00:39:04 DEBUG }, 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "value": [ 00:39:04 DEBUG "result": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG { 00:39:04 DEBUG "0.00033333333333350387" 00:39:04 DEBUG "metric": { 00:39:04 DEBUG ] 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG } 00:39:04 DEBUG }, 00:39:04 DEBUG ] 00:39:04 DEBUG "value": [ 00:39:04 DEBUG } 00:39:04 DEBUG 00:39:04 DEBUG } 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "ds-idrepo-0" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "instance": "10.142.0.107:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "instance": "10.142.0.43:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "14326.4" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "5942272000" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "ds-cts-1" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "2298.5333333333333" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "6.39258829715668" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0.052" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "7071056.317616528" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "2319.8333333333335" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "311973535.91048497" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "instance": "10.142.0.107:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "instance": "10.142.0.43:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "15.859123757205328" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG }, 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "ds-idrepo-0" 00:39:04 DEBUG "pod": "ds-cts-1" 00:39:04 DEBUG }, 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG "0.00006666666666666667" 00:39:04 DEBUG ] 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "ds-idrepo-1" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "ds-idrepo-0" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "0" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "instance": "10.142.0.107:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "instance": "10.142.0.43:9100" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "5439.733333333334" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713137944" 00:39:04 INFO [http_cmd]: http status code OK 00:39:04 DEBUG --- status code --- 00:39:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:39:04 DEBUG --- http response --- 00:39:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:39:04 DEBUG ----- output ----- 00:39:04 DEBUG { 00:39:04 DEBUG "status": "success", 00:39:04 DEBUG "data": { 00:39:04 DEBUG "resultType": "vector", 00:39:04 DEBUG "result": [ 00:39:04 DEBUG { 00:39:04 DEBUG "metric": { 00:39:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG [...] 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG 00:39:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:39:04 DEBUG }, 00:39:04 DEBUG "value": [ 00:39:04 DEBUG 1713137944, 00:39:04 DEBUG "2314.3" 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG ] 00:39:04 DEBUG } 00:39:04 DEBUG } 00:39:20 INFO 00:39:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:39:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:39:20 INFO [loop_until]: OK (rc = 0) 00:39:20 DEBUG --- stdout --- 00:39:20 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12659m 8616Mi am-7d597cf7d6-8vr8f 11791m 8933Mi am-7d597cf7d6-9wz9h 15498m 8828Mi ds-cts-0 11572m 8141Mi ds-cts-1 12101m 8496Mi ds-cts-2 11786m 8201Mi ds-idrepo-0 13016m 4577Mi ds-idrepo-1 2017m 4303Mi ds-idrepo-2 1949m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 3m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6407m 5572Mi 00:39:20 DEBUG --- stderr --- 00:39:20 DEBUG 00:39:25 INFO 00:39:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:39:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:39:25 INFO [loop_until]: OK (rc = 0) 00:39:25 DEBUG --- stdout --- 00:39:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4871Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11391m 38% 11076Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13076m 43% 10691Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17076m 57% 3036Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2238m 7% 6823Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11636m 38% 12310Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2934Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12431m 41% 10411Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2026m 6% 5973Mi 5% gke-xlou-cdm-ds-0f601751-j488 11893m 39% 10717Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2241Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17198m 57% 10433Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13414m 44% 6506Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6566m 41% 9315Mi 15% 00:39:25 DEBUG --- stderr --- 00:39:25 DEBUG 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138004" 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG ----- output ----- 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG { 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG { 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG "data": { 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG "data": { 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:40:04 DEBUG { 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "instance": "10.142.0.107:9100" 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "instance": "10.142.0.107:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "data": { 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG "result": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG { 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "metric": { 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG "pod": "ds-cts-2" 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG }, 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "instance": "10.142.0.43:9100" 00:40:04 DEBUG 00:40:04 DEBUG "instance": "10.142.0.43:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG [...] 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "13378436096" 00:40:04 DEBUG 00:40:04 DEBUG "0.00033333333333350387" 00:40:04 DEBUG ] 00:40:04 DEBUG 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG }, 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG "value": [ 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG } 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "2283.9333333333334" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "413084.5" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138004" 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "instance": "10.142.0.107:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "instance": "10.142.0.43:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "14663.43333333333" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 ERROR [http_cmd]: ERROR 00:40:04 ERROR --- status code --- 00:40:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:40:04 ERROR --- http response --- 00:40:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:40:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "ds-idrepo-0" 00:40:04 DEBUG }, 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "ds-idrepo-0" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "ds-idrepo-0" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "5937823744" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "ds-cts-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "2282.9666666666667" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "6.416687415795839" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0.052" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-cts-2" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "2304.5333333333333" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "instance": "10.142.0.107:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "instance": "10.142.0.43:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-cts-2" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "4614.121152658165" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-cts-2" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0.000026817834950402543" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:40:04 DEBUG "pod": "ds-idrepo-0" 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG }, 00:40:04 DEBUG { 00:40:04 DEBUG "value": [ 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "data": { 00:40:04 DEBUG "0" 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG ] 00:40:04 DEBUG "result": [ 00:40:04 DEBUG } 00:40:04 DEBUG { 00:40:04 DEBUG ] 00:40:04 DEBUG "metric": { 00:40:04 DEBUG } 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG } 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "ds-cts-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0.000029874999999999998" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "ds-idrepo-1" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "ds-idrepo-0" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "0" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138004" 00:40:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138004" 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "2289.866666666667" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:04 INFO [http_cmd]: http status code OK 00:40:04 DEBUG --- status code --- 00:40:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:40:04 DEBUG --- http response --- 00:40:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:40:04 DEBUG ----- output ----- 00:40:04 DEBUG { 00:40:04 DEBUG "status": "success", 00:40:04 DEBUG "data": { 00:40:04 DEBUG "resultType": "vector", 00:40:04 DEBUG "result": [ 00:40:04 DEBUG { 00:40:04 DEBUG "metric": { 00:40:04 DEBUG "instance": "10.142.0.107:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG [...] 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG 00:40:04 DEBUG "instance": "10.142.0.43:9100" 00:40:04 DEBUG }, 00:40:04 DEBUG "value": [ 00:40:04 DEBUG 1713138004, 00:40:04 DEBUG "7359.466666666666" 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG ] 00:40:04 DEBUG } 00:40:04 DEBUG } 00:40:20 INFO 00:40:20 INFO [loop_until]: kubectl --namespace=xlou top pods 00:40:20 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:40:21 INFO [loop_until]: OK (rc = 0) 00:40:21 DEBUG --- stdout --- 00:40:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12808m 8614Mi am-7d597cf7d6-8vr8f 11810m 8935Mi am-7d597cf7d6-9wz9h 16082m 8822Mi ds-cts-0 11899m 8116Mi ds-cts-1 11868m 8498Mi ds-cts-2 12304m 8188Mi ds-idrepo-0 13315m 4577Mi ds-idrepo-1 2013m 4303Mi ds-idrepo-2 1949m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6387m 5573Mi 00:40:21 DEBUG --- stderr --- 00:40:21 DEBUG 00:40:25 INFO 00:40:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:40:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:40:25 INFO [loop_until]: OK (rc = 0) 00:40:25 DEBUG --- stdout --- 00:40:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4874Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12172m 40% 11084Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12959m 43% 10688Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17560m 58% 3049Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2203m 7% 6829Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11790m 39% 12289Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12484m 41% 10415Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2048m 6% 5980Mi 5% gke-xlou-cdm-ds-0f601751-j488 12078m 40% 10719Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2240Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17042m 57% 10418Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13328m 44% 6497Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6539m 41% 9310Mi 15% 00:40:25 DEBUG --- stderr --- 00:40:25 DEBUG 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG ----- output ----- 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG { 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "data": { 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "resultType": "vector", 00:41:04 ERROR [http_cmd]: ERROR 00:41:04 DEBUG "result": [ 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:41:04 DEBUG "result": [ 00:41:04 DEBUG "result": [ 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 ERROR --- status code --- 00:41:04 DEBUG { 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG { 00:41:04 DEBUG { 00:41:04 DEBUG --- status code --- 00:41:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:41:04 DEBUG "metric": { 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 ERROR --- http response --- 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG "instance": "10.142.0.107:9100" 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG }, 00:41:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:41:04 DEBUG }, 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG }, 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG "value": [ 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:41:04 DEBUG "value": [ 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG "value": [ 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG { 00:41:04 DEBUG 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG 00:41:04 DEBUG { 00:41:04 DEBUG 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG "data": { 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG "data": { 00:41:04 DEBUG [...] 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG 00:41:04 DEBUG "result": [ 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG { 00:41:04 DEBUG "result": [ 00:41:04 DEBUG 00:41:04 DEBUG { 00:41:04 DEBUG { 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG { 00:41:04 DEBUG 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG 00:41:04 DEBUG "pod": "ds-idrepo-0" 00:41:04 DEBUG "data": { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "instance": "10.142.0.43:9100" 00:41:04 DEBUG "instance": "10.142.0.107:9100" 00:41:04 DEBUG "data": { 00:41:04 DEBUG "pod": "ds-idrepo-0" 00:41:04 DEBUG }, 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG }, 00:41:04 DEBUG }, 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG "result": [ 00:41:04 DEBUG }, 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG "value": [ 00:41:04 DEBUG "value": [ 00:41:04 DEBUG "result": [ 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG { 00:41:04 DEBUG "value": [ 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG 00:41:04 DEBUG { 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0" 00:41:04 DEBUG "metric": { 00:41:04 DEBUG 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG "0.00040000000000001514" 00:41:04 DEBUG 00:41:04 DEBUG "metric": { 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG "0" 00:41:04 DEBUG ] 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG 00:41:04 DEBUG ] 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG 00:41:04 DEBUG "instance": "10.142.0.107:9100" 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG }, 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG [...] 00:41:04 DEBUG }, 00:41:04 DEBUG } 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ] 00:41:04 DEBUG "value": [ 00:41:04 DEBUG [...] 00:41:04 DEBUG ] 00:41:04 DEBUG { 00:41:04 DEBUG 00:41:04 DEBUG "value": [ 00:41:04 DEBUG ] 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG { 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG "data": { 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "instance": "10.142.0.43:9100" 00:41:04 DEBUG 00:41:04 DEBUG "data": { 00:41:04 DEBUG [...] 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG "result": [ 00:41:04 DEBUG }, 00:41:04 DEBUG [...] 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG 00:41:04 DEBUG }, 00:41:04 DEBUG { 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG "result": [ 00:41:04 DEBUG 00:41:04 DEBUG "value": [ 00:41:04 DEBUG "metric": { 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG 00:41:04 DEBUG { 00:41:04 DEBUG 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG "14657.53333333333" 00:41:04 DEBUG 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG "0" 00:41:04 DEBUG }, 00:41:04 DEBUG ] 00:41:04 DEBUG "instance": "10.142.0.43:9100" 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG }, 00:41:04 DEBUG ] 00:41:04 DEBUG "value": [ 00:41:04 DEBUG } 00:41:04 DEBUG }, 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG ] 00:41:04 DEBUG "value": [ 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG ] 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG 00:41:04 DEBUG "412994" 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG "13378933760" 00:41:04 DEBUG 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG [...] 00:41:04 DEBUG ] 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG [...] 00:41:04 DEBUG ] 00:41:04 DEBUG 00:41:04 DEBUG ] 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG "pod": "ds-idrepo-0" 00:41:04 DEBUG }, 00:41:04 DEBUG 00:41:04 DEBUG } 00:41:04 DEBUG "value": [ 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG }, 00:41:04 DEBUG "0" 00:41:04 DEBUG "value": [ 00:41:04 DEBUG ] 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG } 00:41:04 DEBUG "2300.6" 00:41:04 DEBUG ] 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "5939490816" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "ds-cts-1" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "2277.3" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "6.469567334707353" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "2301.0666666666666" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0.052" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "3415969.3802480167" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "6.885109244107428" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "instance": "10.142.0.107:9100" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "instance": "10.142.0.43:9100" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0.000666666666666658" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "256876925.20601538" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "ds-idrepo-0" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "ds-cts-1" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0.000033333333333333335" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "ds-idrepo-1" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "ds-idrepo-0" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "0" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138064" 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138064" 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "2296.4" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:04 INFO [http_cmd]: http status code OK 00:41:04 DEBUG --- status code --- 00:41:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:41:04 DEBUG --- http response --- 00:41:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:41:04 DEBUG ----- output ----- 00:41:04 DEBUG { 00:41:04 DEBUG "status": "success", 00:41:04 DEBUG "data": { 00:41:04 DEBUG "resultType": "vector", 00:41:04 DEBUG "result": [ 00:41:04 DEBUG { 00:41:04 DEBUG "metric": { 00:41:04 DEBUG "instance": "10.142.0.107:9100" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG [...] 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG 00:41:04 DEBUG "instance": "10.142.0.43:9100" 00:41:04 DEBUG }, 00:41:04 DEBUG "value": [ 00:41:04 DEBUG 1713138064, 00:41:04 DEBUG "5189.866666666667" 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG ] 00:41:04 DEBUG } 00:41:04 DEBUG } 00:41:21 INFO 00:41:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:41:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:41:21 INFO [loop_until]: OK (rc = 0) 00:41:21 DEBUG --- stdout --- 00:41:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12475m 8693Mi am-7d597cf7d6-8vr8f 11445m 8936Mi am-7d597cf7d6-9wz9h 15556m 8891Mi ds-cts-0 11430m 8144Mi ds-cts-1 11974m 8488Mi ds-cts-2 12466m 8191Mi ds-idrepo-0 13071m 4577Mi ds-idrepo-1 2053m 4303Mi ds-idrepo-2 1908m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1014Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6440m 5574Mi 00:41:21 DEBUG --- stderr --- 00:41:21 DEBUG 00:41:25 INFO 00:41:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:41:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:41:25 INFO [loop_until]: OK (rc = 0) 00:41:25 DEBUG --- stdout --- 00:41:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4879Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12284m 41% 11081Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12965m 43% 10766Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17605m 58% 3051Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2251m 7% 6830Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11692m 39% 12298Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2937Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12402m 41% 10409Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2060m 6% 5967Mi 5% gke-xlou-cdm-ds-0f601751-j488 12131m 40% 10719Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2255Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17073m 57% 10493Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13509m 45% 6499Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6680m 42% 9327Mi 15% 00:41:25 DEBUG --- stderr --- 00:41:25 DEBUG 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "ds-idrepo-0" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "instance": "10.142.0.107:9100" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "instance": "10.142.0.43:9100" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "13379537920" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "ds-idrepo-0" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 DEBUG "data": { 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG { 00:42:04 DEBUG "result": [ 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG { 00:42:04 DEBUG "data": { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:42:04 DEBUG "result": [ 00:42:04 DEBUG }, 00:42:04 DEBUG { 00:42:04 DEBUG "value": [ 00:42:04 DEBUG "metric": { 00:42:04 DEBUG 00:42:04 DEBUG "instance": "10.142.0.107:9100" 00:42:04 DEBUG 00:42:04 DEBUG }, 00:42:04 DEBUG 00:42:04 DEBUG "value": [ 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 00:42:04 ERROR [http_cmd]: ERROR 00:42:04 DEBUG 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG }, 00:42:04 ERROR --- status code --- 00:42:04 DEBUG 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG "value": [ 00:42:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:42:04 DEBUG 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG 1713138124, 00:42:04 ERROR --- http response --- 00:42:04 DEBUG "instance": "10.142.0.43:9100" 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:42:04 DEBUG "0" 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:42:04 DEBUG }, 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG ] 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG --- status code --- 00:42:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:42:04 DEBUG "value": [ 00:42:04 DEBUG { 00:42:04 DEBUG } 00:42:04 DEBUG { 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG ] 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG "14746.266666666665" 00:42:04 DEBUG "data": { 00:42:04 DEBUG } 00:42:04 DEBUG "data": { 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:04 DEBUG ] 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG } 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG } 00:42:04 DEBUG "result": [ 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG { 00:42:04 DEBUG ] 00:42:04 DEBUG { 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG } 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "data": { 00:42:04 DEBUG "data": { 00:42:04 DEBUG } 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG "instance": "10.142.0.107:9100" 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG }, 00:42:04 DEBUG }, 00:42:04 DEBUG "result": [ 00:42:04 DEBUG "result": [ 00:42:04 DEBUG "value": [ 00:42:04 DEBUG "value": [ 00:42:04 DEBUG { 00:42:04 DEBUG { 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG }, 00:42:04 DEBUG }, 00:42:04 DEBUG [...] 00:42:04 DEBUG [...] 00:42:04 DEBUG "value": [ 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG "instance": "10.142.0.43:9100" 00:42:04 DEBUG [...] 00:42:04 DEBUG [...] 00:42:04 DEBUG }, 00:42:04 DEBUG }, 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "value": [ 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "413157.5" 00:42:04 DEBUG "0.00036666666666652263" 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG "pod": "ds-idrepo-0" 00:42:04 DEBUG ] 00:42:04 DEBUG ] 00:42:04 DEBUG }, 00:42:04 DEBUG }, 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG "value": [ 00:42:04 DEBUG "value": [ 00:42:04 DEBUG ] 00:42:04 DEBUG ] 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG "2291.866666666667" 00:42:04 DEBUG "0" 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "ds-cts-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "5941149696" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "2292" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0.052" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "2298.633333333333" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "6.439110357738877" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138124" 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG { 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0.000011162800687294798" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "instance": "10.142.0.107:9100" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "instance": "10.142.0.43:9100" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "ds-idrepo-0" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "ds-cts-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0.00031258333333333333" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "ds-idrepo-1" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "ds-idrepo-0" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "0" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138124" 00:42:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138124" 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "2291.766666666667" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:04 INFO [http_cmd]: http status code OK 00:42:04 DEBUG --- status code --- 00:42:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:04 DEBUG --- http response --- 00:42:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:04 DEBUG ----- output ----- 00:42:04 DEBUG { 00:42:04 DEBUG "status": "success", 00:42:04 DEBUG "data": { 00:42:04 DEBUG "resultType": "vector", 00:42:04 DEBUG "result": [ 00:42:04 DEBUG { 00:42:04 DEBUG "metric": { 00:42:04 DEBUG "instance": "10.142.0.107:9100" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG [...] 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG 00:42:04 DEBUG "instance": "10.142.0.43:9100" 00:42:04 DEBUG }, 00:42:04 DEBUG "value": [ 00:42:04 DEBUG 1713138124, 00:42:04 DEBUG "5521.5" 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG ] 00:42:04 DEBUG } 00:42:04 DEBUG } 00:42:21 INFO 00:42:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:42:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:21 INFO [loop_until]: OK (rc = 0) 00:42:21 DEBUG --- stdout --- 00:42:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12226m 8632Mi am-7d597cf7d6-8vr8f 11868m 8933Mi am-7d597cf7d6-9wz9h 16001m 8821Mi ds-cts-0 11659m 8119Mi ds-cts-1 12225m 8477Mi ds-cts-2 11911m 8188Mi ds-idrepo-0 13204m 4577Mi ds-idrepo-1 2005m 4303Mi ds-idrepo-2 1906m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6331m 5573Mi 00:42:21 DEBUG --- stderr --- 00:42:21 DEBUG 00:42:25 INFO 00:42:25 INFO [loop_until]: kubectl --namespace=xlou top node 00:42:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:25 INFO [loop_until]: OK (rc = 0) 00:42:25 DEBUG --- stdout --- 00:42:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4886Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12084m 40% 11077Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13044m 43% 10724Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17571m 58% 3040Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2195m 7% 6833Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11654m 39% 12318Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12518m 41% 10414Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2059m 6% 5967Mi 5% gke-xlou-cdm-ds-0f601751-j488 12278m 41% 10721Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2251Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17172m 57% 10425Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13430m 44% 6501Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6620m 41% 9326Mi 15% 00:42:25 DEBUG --- stderr --- 00:42:25 DEBUG 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138184" 00:43:04 ERROR [http_cmd]: ERROR 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 ERROR --- status code --- 00:43:04 DEBUG --- status code --- 00:43:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 ERROR --- http response --- 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:43:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "412914.5" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "ds-idrepo-0" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "data": { 00:43:04 DEBUG "data": { 00:43:04 DEBUG "data": { 00:43:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG "result": [ 00:43:04 DEBUG "result": [ 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-cts-2" 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG "instance": "10.142.0.107:9100" 00:43:04 DEBUG "instance": "10.142.0.107:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG [...] 00:43:04 DEBUG [...] 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:43:04 DEBUG "pod": "ds-idrepo-0" 00:43:04 DEBUG "instance": "10.142.0.43:9100" 00:43:04 DEBUG "instance": "10.142.0.43:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG "0" 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG "13380180992" 00:43:04 DEBUG "0.0004666666666665264" 00:43:04 DEBUG ] 00:43:04 DEBUG ] 00:43:04 DEBUG ] 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG ] 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138184" 00:43:04 DEBUG ] 00:43:04 DEBUG { 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG "data": { 00:43:04 DEBUG } 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "ds-idrepo-0" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "instance": "10.142.0.107:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "instance": "10.142.0.43:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "14812.899999999998" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "2292.3" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "ds-cts-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "5939499008" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "2293.5" 00:43:04 DEBUG "0.052" 00:43:04 DEBUG ] 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "2285.5" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "6.526294777826128" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "instance": "10.142.0.107:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "instance": "10.142.0.43:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0.00033333333333333365" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-cts-2" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-cts-2" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-cts-2" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "ds-idrepo-0" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "ds-idrepo-0" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "ds-idrepo-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "ds-cts-1" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "0.00023270833333333333" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "2290.733333333333" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138184" 00:43:04 INFO [http_cmd]: http status code OK 00:43:04 DEBUG --- status code --- 00:43:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:04 DEBUG --- http response --- 00:43:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:04 DEBUG ----- output ----- 00:43:04 DEBUG { 00:43:04 DEBUG "status": "success", 00:43:04 DEBUG "data": { 00:43:04 DEBUG "resultType": "vector", 00:43:04 DEBUG "result": [ 00:43:04 DEBUG { 00:43:04 DEBUG "metric": { 00:43:04 DEBUG "instance": "10.142.0.107:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG [...] 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG 00:43:04 DEBUG "instance": "10.142.0.43:9100" 00:43:04 DEBUG }, 00:43:04 DEBUG "value": [ 00:43:04 DEBUG 1713138184, 00:43:04 DEBUG "5687.833333333333" 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG ] 00:43:04 DEBUG } 00:43:04 DEBUG } 00:43:21 INFO 00:43:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:43:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:43:21 INFO [loop_until]: OK (rc = 0) 00:43:21 DEBUG --- stdout --- 00:43:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12524m 8656Mi am-7d597cf7d6-8vr8f 11933m 8933Mi am-7d597cf7d6-9wz9h 15985m 8823Mi ds-cts-0 11787m 8132Mi ds-cts-1 11836m 8503Mi ds-cts-2 11809m 8210Mi ds-idrepo-0 13274m 4577Mi ds-idrepo-1 1962m 4303Mi ds-idrepo-2 1863m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6528m 5573Mi 00:43:21 DEBUG --- stderr --- 00:43:21 DEBUG 00:43:26 INFO 00:43:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:43:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:43:26 INFO [loop_until]: OK (rc = 0) 00:43:26 DEBUG --- stdout --- 00:43:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11921m 39% 11081Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13109m 43% 10723Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17026m 57% 3037Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2169m 7% 6830Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11714m 39% 12333Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12099m 40% 10410Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2037m 6% 5966Mi 5% gke-xlou-cdm-ds-0f601751-j488 12223m 40% 10723Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2238Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16975m 56% 10423Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13390m 44% 6500Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6605m 41% 9313Mi 15% 00:43:26 DEBUG --- stderr --- 00:43:26 DEBUG 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 ERROR [http_cmd]: ERROR 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 ERROR --- status code --- 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG --- status code --- 00:44:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 ERROR --- http response --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- http response --- 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG { 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG { 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "data": { 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG "data": { 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG { 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG "result": [ 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG "result": [ 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138244" 00:44:04 DEBUG { 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG "data": { 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "data": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG "metric": { 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG }, 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG { 00:44:04 DEBUG "result": [ 00:44:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:44:04 DEBUG }, 00:44:04 DEBUG "result": [ 00:44:04 DEBUG "value": [ 00:44:04 DEBUG }, 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG { 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG { 00:44:04 DEBUG 00:44:04 DEBUG "value": [ 00:44:04 DEBUG "data": { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG "metric": { 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "instance": "10.142.0.107:9100" 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "result": [ 00:44:04 DEBUG }, 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG }, 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG { 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG "metric": { 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG }, 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG 00:44:04 DEBUG "value": [ 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG [...] 00:44:04 DEBUG }, 00:44:04 DEBUG "pod": "ds-idrepo-0" 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG 00:44:04 DEBUG "value": [ 00:44:04 DEBUG }, 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG 00:44:04 DEBUG "413173" 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG [...] 00:44:04 DEBUG "pod": "ds-idrepo-0" 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "2295.2666666666664" 00:44:04 DEBUG "instance": "10.142.0.43:9100" 00:44:04 DEBUG ] 00:44:04 DEBUG "0" 00:44:04 DEBUG 00:44:04 DEBUG }, 00:44:04 DEBUG "0" 00:44:04 DEBUG ] 00:44:04 DEBUG }, 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG 00:44:04 DEBUG "value": [ 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG "value": [ 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG "pod": "ds-idrepo-0" 00:44:04 DEBUG "0" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG "13380887552" 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG }, 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG "value": [ 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG ] 00:44:04 DEBUG ] 00:44:04 DEBUG "0" 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "instance": "10.142.0.107:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "instance": "10.142.0.43:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0.0002666666666669926" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG "result": [ 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG { 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG "metric": { 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "ds-cts-1" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "instance": "10.142.0.107:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "instance": "10.142.0.43:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "14700.899999999998" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "6172495872" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "2301.4333333333334" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0.052" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "2295.6" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "6.462790422280327" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "16.059774683543907" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "9120.26841543389" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "instance": "10.142.0.107:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "instance": "10.142.0.43:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0.0003333333333333558" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "10769662.71160592" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "ds-idrepo-0" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "ds-idrepo-0" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "ds-idrepo-1" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "ds-cts-1" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "0.00013333333333333334" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "2304.5666666666666" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138244" 00:44:04 INFO [http_cmd]: http status code OK 00:44:04 DEBUG --- status code --- 00:44:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:04 DEBUG --- http response --- 00:44:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:04 DEBUG ----- output ----- 00:44:04 DEBUG { 00:44:04 DEBUG "status": "success", 00:44:04 DEBUG "data": { 00:44:04 DEBUG "resultType": "vector", 00:44:04 DEBUG "result": [ 00:44:04 DEBUG { 00:44:04 DEBUG "metric": { 00:44:04 DEBUG "instance": "10.142.0.107:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG [...] 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG 00:44:04 DEBUG "instance": "10.142.0.43:9100" 00:44:04 DEBUG }, 00:44:04 DEBUG "value": [ 00:44:04 DEBUG 1713138244, 00:44:04 DEBUG "5513.466666666666" 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG ] 00:44:04 DEBUG } 00:44:04 DEBUG } 00:44:21 INFO 00:44:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:44:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:44:21 INFO [loop_until]: OK (rc = 0) 00:44:21 DEBUG --- stdout --- 00:44:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12261m 8629Mi am-7d597cf7d6-8vr8f 11572m 8933Mi am-7d597cf7d6-9wz9h 16140m 8882Mi ds-cts-0 11803m 8208Mi ds-cts-1 11787m 8493Mi ds-cts-2 12045m 8212Mi ds-idrepo-0 13107m 4577Mi ds-idrepo-1 1956m 4303Mi ds-idrepo-2 1931m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6293m 5793Mi 00:44:21 DEBUG --- stderr --- 00:44:21 DEBUG 00:44:26 INFO 00:44:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:44:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:44:26 INFO [loop_until]: OK (rc = 0) 00:44:26 DEBUG --- stdout --- 00:44:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4870Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12338m 41% 11079Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13057m 43% 10710Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17493m 58% 3045Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2175m 7% 6832Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11556m 38% 12375Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2934Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12449m 41% 10407Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2043m 6% 5979Mi 5% gke-xlou-cdm-ds-0f601751-j488 11884m 39% 10722Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2237Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17206m 57% 10481Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13406m 44% 6502Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6545m 41% 9541Mi 16% 00:44:26 DEBUG --- stderr --- 00:44:26 DEBUG 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 ERROR [http_cmd]: ERROR 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 ERROR --- status code --- 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 ERROR --- http response --- 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "data": { 00:45:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG "resultType": "vector", 00:45:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-cts-2" 00:45:04 DEBUG }, 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG 00:45:04 DEBUG { 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG "pod": "ds-idrepo-0" 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG }, 00:45:04 DEBUG "data": { 00:45:04 DEBUG "pod": "ds-idrepo-0" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG }, 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG "result": [ 00:45:04 DEBUG "value": [ 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0" 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG { 00:45:04 DEBUG { 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG "0" 00:45:04 DEBUG ] 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "0" 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG "data": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:45:04 DEBUG ] 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG }, 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG "result": [ 00:45:04 DEBUG ----- output ----- 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 DEBUG "value": [ 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 DEBUG { 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 DEBUG "metric": { 00:45:04 DEBUG 00:45:04 DEBUG "data": { 00:45:04 DEBUG } 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG }, 00:45:04 DEBUG 00:45:04 DEBUG "result": [ 00:45:04 DEBUG "value": [ 00:45:04 DEBUG [...] 00:45:04 DEBUG { 00:45:04 DEBUG 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG 00:45:04 DEBUG "metric": { 00:45:04 DEBUG 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG 00:45:04 DEBUG "instance": "10.142.0.107:9100" 00:45:04 DEBUG 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG 00:45:04 DEBUG }, 00:45:04 DEBUG [...] 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG }, 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138304" 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "value": [ 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG { 00:45:04 DEBUG 00:45:04 DEBUG "pod": "ds-cts-1" 00:45:04 DEBUG "413656.5" 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG [...] 00:45:04 DEBUG }, 00:45:04 DEBUG ] 00:45:04 DEBUG "data": { 00:45:04 DEBUG 00:45:04 DEBUG "value": [ 00:45:04 DEBUG } 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG ] 00:45:04 DEBUG "result": [ 00:45:04 DEBUG 00:45:04 DEBUG "0" 00:45:04 DEBUG } 00:45:04 DEBUG { 00:45:04 DEBUG "instance": "10.142.0.43:9100" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG "metric": { 00:45:04 DEBUG }, 00:45:04 DEBUG } 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG "value": [ 00:45:04 DEBUG ] 00:45:04 DEBUG }, 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG } 00:45:04 DEBUG "value": [ 00:45:04 DEBUG "13381643264" 00:45:04 DEBUG } 00:45:04 DEBUG 00:45:04 DEBUG ] 00:45:04 DEBUG 00:45:04 DEBUG } 00:45:04 DEBUG 00:45:04 DEBUG ] 00:45:04 DEBUG [...] 00:45:04 DEBUG } 00:45:04 DEBUG 00:45:04 DEBUG } 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "ds-idrepo-0" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "2322.4" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "instance": "10.142.0.107:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "instance": "10.142.0.43:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0.00043333333333303397" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "instance": "10.142.0.107:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "instance": "10.142.0.43:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "14985.93333333333" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "2322.5666666666666" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "2291" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0.052" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "6171017216" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "6.437005390914983" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-cts-2" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "15.97070812256601" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-cts-2" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "915342.5199139484" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "instance": "10.142.0.107:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "instance": "10.142.0.43:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0.00033333333333333365" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-cts-2" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "9162355.950927379" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "ds-idrepo-0" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "ds-idrepo-0" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "ds-idrepo-1" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "ds-cts-1" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "0.0004333333333333333" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "2308.4333333333334" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138304" 00:45:04 INFO [http_cmd]: http status code OK 00:45:04 DEBUG --- status code --- 00:45:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:04 DEBUG --- http response --- 00:45:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:04 DEBUG ----- output ----- 00:45:04 DEBUG { 00:45:04 DEBUG "status": "success", 00:45:04 DEBUG "data": { 00:45:04 DEBUG "resultType": "vector", 00:45:04 DEBUG "result": [ 00:45:04 DEBUG { 00:45:04 DEBUG "metric": { 00:45:04 DEBUG "instance": "10.142.0.107:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG [...] 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG 00:45:04 DEBUG "instance": "10.142.0.43:9100" 00:45:04 DEBUG }, 00:45:04 DEBUG "value": [ 00:45:04 DEBUG 1713138304, 00:45:04 DEBUG "7111.166666666667" 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG ] 00:45:04 DEBUG } 00:45:04 DEBUG } 00:45:21 INFO 00:45:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:45:21 INFO [loop_until]: OK (rc = 0) 00:45:21 DEBUG --- stdout --- 00:45:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12936m 8621Mi am-7d597cf7d6-8vr8f 11652m 8936Mi am-7d597cf7d6-9wz9h 15686m 8853Mi ds-cts-0 11567m 8184Mi ds-cts-1 12059m 8489Mi ds-cts-2 12283m 8216Mi ds-idrepo-0 13617m 4577Mi ds-idrepo-1 2034m 4303Mi ds-idrepo-2 1898m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 6m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6252m 5794Mi 00:45:21 DEBUG --- stderr --- 00:45:21 DEBUG 00:45:26 INFO 00:45:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:45:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:45:26 INFO [loop_until]: OK (rc = 0) 00:45:26 DEBUG --- stdout --- 00:45:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4875Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12126m 40% 11088Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12643m 42% 10708Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17553m 58% 3053Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2211m 7% 6829Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11780m 39% 12388Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12522m 41% 10413Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2053m 6% 5980Mi 5% gke-xlou-cdm-ds-0f601751-j488 12207m 40% 10727Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2250Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17163m 57% 10454Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13485m 45% 6496Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6568m 41% 9542Mi 16% 00:45:26 DEBUG --- stderr --- 00:45:26 DEBUG 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG --- status code --- 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG "instance": "10.142.0.107:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-idrepo-0" 00:46:04 DEBUG "instance": "10.142.0.43:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG "13382210560" 00:46:04 DEBUG ] 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 ERROR [http_cmd]: ERROR 00:46:04 ERROR --- status code --- 00:46:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:46:04 ERROR --- http response --- 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG --- status code --- 00:46:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG "result": [ 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG "data": { 00:46:04 DEBUG }, 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "value": [ 00:46:04 DEBUG "result": [ 00:46:04 DEBUG 00:46:04 DEBUG { 00:46:04 DEBUG 00:46:04 DEBUG "metric": { 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG [...] 00:46:04 DEBUG }, 00:46:04 DEBUG 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-idrepo-0" 00:46:04 DEBUG 00:46:04 DEBUG }, 00:46:04 DEBUG [...] 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG 00:46:04 DEBUG "0" 00:46:04 DEBUG 00:46:04 DEBUG ] 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138364" 00:46:04 DEBUG }, 00:46:04 DEBUG ] 00:46:04 DEBUG "value": [ 00:46:04 DEBUG } 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG } 00:46:04 DEBUG "2270.5666666666666" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG }, 00:46:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:46:04 DEBUG }, 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-idrepo-0" 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG "pod": "ds-cts-1" 00:46:04 DEBUG }, 00:46:04 DEBUG "pod": "ds-cts-0" 00:46:04 DEBUG }, 00:46:04 DEBUG { 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "data": { 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "413610.5" 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG "20691601.002506264" 00:46:04 DEBUG ] 00:46:04 DEBUG "result": [ 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG { 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG "metric": { 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG "instance": "10.142.0.107:9100" 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 DEBUG }, 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "instance": "10.142.0.43:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0.00036666666666652263" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0.053" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "instance": "10.142.0.107:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "instance": "10.142.0.43:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "14355.533333333331" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "2269.7666666666664" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "2287.6" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "6175064064" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "6.406579694594975" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "instance": "10.142.0.107:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "instance": "10.142.0.43:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-cts-0" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "52309461.65413534" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-idrepo-0" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-idrepo-0" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "ds-idrepo-1" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "ds-cts-1" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "0.0002666666666666667" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138364" 00:46:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138364" 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "2296.733333333333" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:04 INFO [http_cmd]: http status code OK 00:46:04 DEBUG --- status code --- 00:46:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:04 DEBUG --- http response --- 00:46:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:04 DEBUG ----- output ----- 00:46:04 DEBUG { 00:46:04 DEBUG "status": "success", 00:46:04 DEBUG "data": { 00:46:04 DEBUG "resultType": "vector", 00:46:04 DEBUG "result": [ 00:46:04 DEBUG { 00:46:04 DEBUG "metric": { 00:46:04 DEBUG "instance": "10.142.0.107:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG [...] 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG 00:46:04 DEBUG "instance": "10.142.0.43:9100" 00:46:04 DEBUG }, 00:46:04 DEBUG "value": [ 00:46:04 DEBUG 1713138364, 00:46:04 DEBUG "5484.166666666666" 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG ] 00:46:04 DEBUG } 00:46:04 DEBUG } 00:46:21 INFO 00:46:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:46:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:46:21 INFO [loop_until]: OK (rc = 0) 00:46:21 DEBUG --- stdout --- 00:46:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12455m 8625Mi am-7d597cf7d6-8vr8f 11949m 8933Mi am-7d597cf7d6-9wz9h 15847m 8849Mi ds-cts-0 11695m 8215Mi ds-cts-1 12538m 8511Mi ds-cts-2 12251m 8191Mi ds-idrepo-0 13455m 4577Mi ds-idrepo-1 2007m 4303Mi ds-idrepo-2 1885m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1014Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6503m 5798Mi 00:46:21 DEBUG --- stderr --- 00:46:21 DEBUG 00:46:26 INFO 00:46:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:46:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:46:26 INFO [loop_until]: OK (rc = 0) 00:46:26 DEBUG --- stdout --- 00:46:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4889Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11886m 39% 11083Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12947m 43% 10701Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17585m 58% 3071Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2238m 7% 6831Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11933m 39% 12362Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12537m 41% 10405Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2018m 6% 5982Mi 5% gke-xlou-cdm-ds-0f601751-j488 12318m 41% 10697Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2240Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17045m 57% 10438Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13378m 44% 6508Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6577m 41% 9546Mi 16% 00:46:26 DEBUG --- stderr --- 00:46:26 DEBUG 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "2280.366666666667" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 ERROR [http_cmd]: ERROR 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 ERROR --- status code --- 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 ERROR --- http response --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG }, 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 DEBUG "pod": "ds-idrepo-0" 00:47:04 DEBUG "pod": "ds-idrepo-0" 00:47:04 DEBUG }, 00:47:04 DEBUG }, 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG "value": [ 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG "value": [ 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG "0" 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG "0" 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG ] 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:47:04 DEBUG } 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:04 DEBUG ] 00:47:04 DEBUG ] 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG { 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "instance": "10.142.0.107:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "instance": "10.142.0.43:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "13382857728" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "ds-cts-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "ds-idrepo-0" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG { 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "data": { 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG "result": [ 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG { 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:04 DEBUG "instance": "10.142.0.107:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG }, 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG "value": [ 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG [...] 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:04 DEBUG "instance": "10.142.0.43:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG }, 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG "value": [ 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "2279.6666666666665" 00:47:04 DEBUG "414004.5" 00:47:04 DEBUG "0.00040000000000001514" 00:47:04 DEBUG ] 00:47:04 DEBUG ] 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG ] 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0.053" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "instance": "10.142.0.107:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "instance": "10.142.0.43:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "14422.199999999997" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "2293.1" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "6175551488" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "6.416367680254649" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "10097.305440476597" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0.002950251652002716" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "560.961413359811" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "instance": "10.142.0.107:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "instance": "10.142.0.43:9100" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0.000666666666666682" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "ds-idrepo-0" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "ds-idrepo-0" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138424" 00:47:04 INFO [http_cmd]: http status code OK 00:47:04 DEBUG --- status code --- 00:47:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:04 DEBUG --- http response --- 00:47:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:47:04 DEBUG ----- output ----- 00:47:04 DEBUG { 00:47:04 DEBUG "status": "success", 00:47:04 DEBUG "data": { 00:47:04 DEBUG "resultType": "vector", 00:47:04 DEBUG "result": [ 00:47:04 DEBUG { 00:47:04 DEBUG "metric": { 00:47:04 DEBUG "pod": "ds-idrepo-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG [...] 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG 00:47:04 DEBUG "pod": "ds-cts-1" 00:47:04 DEBUG }, 00:47:04 DEBUG "value": [ 00:47:04 DEBUG 1713138424, 00:47:04 DEBUG "0.00006320833333333333" 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG ] 00:47:04 DEBUG } 00:47:04 DEBUG } 00:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138425" 00:47:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138425" 00:47:05 INFO [http_cmd]: http status code OK 00:47:05 DEBUG --- status code --- 00:47:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:05 DEBUG --- http response --- 00:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:05 INFO [http_cmd]: http status code OK 00:47:05 DEBUG ----- output ----- 00:47:05 DEBUG --- status code --- 00:47:05 DEBUG { 00:47:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:05 DEBUG "status": "success", 00:47:05 DEBUG --- http response --- 00:47:05 DEBUG "data": { 00:47:05 DEBUG "resultType": "vector", 00:47:05 DEBUG "result": [ 00:47:05 DEBUG { 00:47:05 DEBUG "metric": { 00:47:05 DEBUG "instance": "10.142.0.107:9100" 00:47:05 DEBUG }, 00:47:05 DEBUG "value": [ 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG [...] 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG "instance": "10.142.0.43:9100" 00:47:05 DEBUG }, 00:47:05 DEBUG "value": [ 00:47:05 DEBUG 1713138425, 00:47:05 DEBUG "5638.7" 00:47:05 DEBUG ] 00:47:05 DEBUG } 00:47:05 DEBUG ] 00:47:05 DEBUG } 00:47:05 DEBUG } 00:47:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:05 DEBUG ----- output ----- 00:47:05 DEBUG { 00:47:05 DEBUG "status": "success", 00:47:05 DEBUG "data": { 00:47:05 DEBUG "resultType": "vector", 00:47:05 DEBUG "result": [ 00:47:05 DEBUG { 00:47:05 DEBUG "metric": { 00:47:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:47:05 DEBUG }, 00:47:05 DEBUG "value": [ 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG [...] 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG 00:47:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:47:05 DEBUG }, 00:47:05 DEBUG "value": [ 00:47:05 DEBUG 1713138425, 00:47:05 DEBUG "2291.766666666667" 00:47:05 DEBUG ] 00:47:05 DEBUG } 00:47:05 DEBUG ] 00:47:05 DEBUG } 00:47:05 DEBUG } 00:47:21 INFO 00:47:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:47:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:47:21 INFO [loop_until]: OK (rc = 0) 00:47:21 DEBUG --- stdout --- 00:47:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12347m 8627Mi am-7d597cf7d6-8vr8f 11625m 8934Mi am-7d597cf7d6-9wz9h 15567m 8826Mi ds-cts-0 11135m 8209Mi ds-cts-1 11923m 8495Mi ds-cts-2 12238m 8191Mi ds-idrepo-0 13748m 4577Mi ds-idrepo-1 2023m 4303Mi ds-idrepo-2 1963m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6506m 5796Mi 00:47:21 DEBUG --- stderr --- 00:47:21 DEBUG 00:47:26 INFO 00:47:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:47:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:47:26 INFO [loop_until]: OK (rc = 0) 00:47:26 DEBUG --- stdout --- 00:47:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4895Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12505m 41% 11087Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13149m 44% 10707Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17040m 57% 3049Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2246m 7% 6832Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11469m 38% 12369Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2936Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12488m 41% 10404Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2019m 6% 5965Mi 5% gke-xlou-cdm-ds-0f601751-j488 11983m 40% 10718Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2251Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16949m 56% 10422Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13419m 44% 6502Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6373m 40% 9543Mi 16% 00:47:26 DEBUG --- stderr --- 00:47:26 DEBUG 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "ds-idrepo-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "ds-cts-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "ds-idrepo-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "ds-idrepo-0" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138484" 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "ds-idrepo-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "ds-idrepo-0" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 ERROR [http_cmd]: ERROR 00:48:04 ERROR --- status code --- 00:48:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:48:04 ERROR --- http response --- 00:48:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:48:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "2306.7666666666664" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "ds-idrepo-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "ds-idrepo-0" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "instance": "10.142.0.107:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "instance": "10.142.0.43:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "13383578624" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "413446.5" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "idm-b85f9d978-rl76f" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "instance": "10.142.0.107:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "instance": "10.142.0.43:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0.00030000000000001136" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "2306.7" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0.053" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "instance": "10.142.0.107:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "instance": "10.142.0.43:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "14394.699999999999" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "data": { 00:48:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG }, 00:48:04 DEBUG "result": [ 00:48:04 DEBUG "value": [ 00:48:04 DEBUG { 00:48:04 DEBUG 00:48:04 DEBUG "metric": { 00:48:04 DEBUG 00:48:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:04 DEBUG 00:48:04 DEBUG }, 00:48:04 DEBUG [...] 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:04 DEBUG [...] 00:48:04 DEBUG }, 00:48:04 DEBUG 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG 00:48:04 DEBUG "0" 00:48:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:04 DEBUG ] 00:48:04 DEBUG }, 00:48:04 DEBUG } 00:48:04 DEBUG "value": [ 00:48:04 DEBUG ] 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG } 00:48:04 DEBUG "2320.5" 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "idm-b85f9d978-rl76f" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "650.50979893911" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138484" 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "6174277632" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "6.42039466602359" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "idm-b85f9d978-rl76f" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0.002967506273225721" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "idm-b85f9d978-rl76f" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "520.407839151288" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "instance": "10.142.0.107:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "instance": "10.142.0.43:9100" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "ds-idrepo-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "ds-idrepo-0" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138484" 00:48:04 INFO [http_cmd]: http status code OK 00:48:04 DEBUG --- status code --- 00:48:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:04 DEBUG --- http response --- 00:48:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:04 DEBUG ----- output ----- 00:48:04 DEBUG { 00:48:04 DEBUG "status": "success", 00:48:04 DEBUG "data": { 00:48:04 DEBUG "resultType": "vector", 00:48:04 DEBUG "result": [ 00:48:04 DEBUG { 00:48:04 DEBUG "metric": { 00:48:04 DEBUG "pod": "ds-idrepo-1" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG [...] 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG 00:48:04 DEBUG "pod": "ds-idrepo-0" 00:48:04 DEBUG }, 00:48:04 DEBUG "value": [ 00:48:04 DEBUG 1713138484, 00:48:04 DEBUG "0" 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG ] 00:48:04 DEBUG } 00:48:04 DEBUG } 00:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138485" 00:48:05 INFO [http_cmd]: http status code OK 00:48:05 DEBUG --- status code --- 00:48:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:05 DEBUG --- http response --- 00:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:48:05 DEBUG ----- output ----- 00:48:05 DEBUG { 00:48:05 DEBUG "status": "success", 00:48:05 DEBUG "data": { 00:48:05 DEBUG "resultType": "vector", 00:48:05 DEBUG "result": [ 00:48:05 DEBUG { 00:48:05 DEBUG "metric": { 00:48:05 DEBUG "pod": "ds-idrepo-1" 00:48:05 DEBUG }, 00:48:05 DEBUG "value": [ 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG [...] 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG "pod": "ds-cts-1" 00:48:05 DEBUG }, 00:48:05 DEBUG "value": [ 00:48:05 DEBUG 1713138485, 00:48:05 DEBUG "0.00013333333333333334" 00:48:05 DEBUG ] 00:48:05 DEBUG } 00:48:05 DEBUG ] 00:48:05 DEBUG } 00:48:05 DEBUG } 00:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138485" 00:48:05 INFO [http_cmd]: http status code OK 00:48:05 DEBUG --- status code --- 00:48:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:05 DEBUG --- http response --- 00:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:05 DEBUG ----- output ----- 00:48:05 DEBUG { 00:48:05 DEBUG "status": "success", 00:48:05 DEBUG "data": { 00:48:05 DEBUG "resultType": "vector", 00:48:05 DEBUG "result": [ 00:48:05 DEBUG { 00:48:05 DEBUG "metric": { 00:48:05 DEBUG "instance": "10.142.0.107:9100" 00:48:05 DEBUG }, 00:48:05 DEBUG "value": [ 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG [...] 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG "instance": "10.142.0.43:9100" 00:48:05 DEBUG }, 00:48:05 DEBUG "value": [ 00:48:05 DEBUG 1713138485, 00:48:05 DEBUG "5526.566666666667" 00:48:05 DEBUG ] 00:48:05 DEBUG } 00:48:05 DEBUG ] 00:48:05 DEBUG } 00:48:05 DEBUG } 00:48:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138485" 00:48:05 INFO [http_cmd]: http status code OK 00:48:05 DEBUG --- status code --- 00:48:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:05 DEBUG --- http response --- 00:48:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:05 DEBUG ----- output ----- 00:48:05 DEBUG { 00:48:05 DEBUG "status": "success", 00:48:05 DEBUG "data": { 00:48:05 DEBUG "resultType": "vector", 00:48:05 DEBUG "result": [ 00:48:05 DEBUG { 00:48:05 DEBUG "metric": { 00:48:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:48:05 DEBUG }, 00:48:05 DEBUG "value": [ 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG [...] 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG 00:48:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:48:05 DEBUG }, 00:48:05 DEBUG "value": [ 00:48:05 DEBUG 1713138485, 00:48:05 DEBUG "2295.733333333333" 00:48:05 DEBUG ] 00:48:05 DEBUG } 00:48:05 DEBUG ] 00:48:05 DEBUG } 00:48:05 DEBUG } 00:48:21 INFO 00:48:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:48:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:48:21 INFO [loop_until]: OK (rc = 0) 00:48:21 DEBUG --- stdout --- 00:48:21 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12688m 8628Mi am-7d597cf7d6-8vr8f 10990m 8937Mi am-7d597cf7d6-9wz9h 15656m 8825Mi ds-cts-0 11094m 8184Mi ds-cts-1 11924m 8512Mi ds-cts-2 12134m 8217Mi ds-idrepo-0 12992m 4577Mi ds-idrepo-1 2044m 4303Mi ds-idrepo-2 2002m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6359m 5795Mi 00:48:21 DEBUG --- stderr --- 00:48:21 DEBUG 00:48:26 INFO 00:48:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:48:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:48:26 INFO [loop_until]: OK (rc = 0) 00:48:26 DEBUG --- stdout --- 00:48:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4880Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12174m 40% 11087Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13202m 44% 10705Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17559m 58% 3032Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2254m 7% 6840Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11566m 38% 12394Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12484m 41% 10409Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2001m 6% 5962Mi 5% gke-xlou-cdm-ds-0f601751-j488 12247m 41% 10733Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2229Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 17079m 57% 10433Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13418m 44% 6503Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6643m 41% 9542Mi 16% 00:48:26 DEBUG --- stderr --- 00:48:26 DEBUG 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "2293.9" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 ERROR [http_cmd]: ERROR 00:49:04 ERROR --- status code --- 00:49:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:49:04 ERROR --- http response --- 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG --- status code --- 00:49:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "instance": "10.142.0.107:9100" 00:49:04 DEBUG "pod": "ds-idrepo-1" 00:49:04 DEBUG }, 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "instance": "10.142.0.43:9100" 00:49:04 DEBUG "pod": "ds-cts-1" 00:49:04 DEBUG }, 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "13384272896" 00:49:04 DEBUG "0" 00:49:04 DEBUG ] 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "ds-idrepo-1" 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "ds-idrepo-1" 00:49:04 DEBUG "pod": "ds-idrepo-1" 00:49:04 DEBUG }, 00:49:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:04 DEBUG }, 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "ds-idrepo-0" 00:49:04 DEBUG 00:49:04 DEBUG "pod": "ds-idrepo-0" 00:49:04 DEBUG "pod": "ds-idrepo-0" 00:49:04 DEBUG }, 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0" 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0" 00:49:04 DEBUG "0" 00:49:04 DEBUG ] 00:49:04 DEBUG "413988" 00:49:04 DEBUG ] 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "ds-cts-1" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "4389.101449275362" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "instance": "10.142.0.107:9100" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "instance": "10.142.0.43:9100" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0.0002666666666669926" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "2293.0333333333333" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0.053" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "instance": "10.142.0.107:9100" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "instance": "10.142.0.43:9100" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "14725.033333333333" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "2271.866666666667" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138544" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "6175240192" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "6.410323142321074" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138544" 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "ds-cts-1" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "35075316.12753623" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: http status code OK 00:49:04 DEBUG --- status code --- 00:49:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:04 DEBUG --- http response --- 00:49:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:04 DEBUG ----- output ----- 00:49:04 DEBUG { 00:49:04 DEBUG "status": "success", 00:49:04 DEBUG "data": { 00:49:04 DEBUG "resultType": "vector", 00:49:04 DEBUG "result": [ 00:49:04 DEBUG { 00:49:04 DEBUG "metric": { 00:49:04 DEBUG "pod": "ds-idrepo-1" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG [...] 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG 00:49:04 DEBUG "pod": "ds-idrepo-0" 00:49:04 DEBUG }, 00:49:04 DEBUG "value": [ 00:49:04 DEBUG 1713138544, 00:49:04 DEBUG "0" 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG ] 00:49:04 DEBUG } 00:49:04 DEBUG } 00:49:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138544" 00:49:05 INFO [http_cmd]: http status code OK 00:49:05 DEBUG --- status code --- 00:49:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:05 DEBUG --- http response --- 00:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:05 DEBUG ----- output ----- 00:49:05 DEBUG { 00:49:05 DEBUG "status": "success", 00:49:05 DEBUG "data": { 00:49:05 DEBUG "resultType": "vector", 00:49:05 DEBUG "result": [ 00:49:05 DEBUG { 00:49:05 DEBUG "metric": { 00:49:05 DEBUG "instance": "10.142.0.107:9100" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG [...] 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG "instance": "10.142.0.43:9100" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 1713138544, 00:49:05 DEBUG "0.00033333333333333365" 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG } 00:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138545" 00:49:05 INFO [http_cmd]: http status code OK 00:49:05 DEBUG --- status code --- 00:49:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:05 DEBUG --- http response --- 00:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:05 DEBUG ----- output ----- 00:49:05 DEBUG { 00:49:05 DEBUG "status": "success", 00:49:05 DEBUG "data": { 00:49:05 DEBUG "resultType": "vector", 00:49:05 DEBUG "result": [ 00:49:05 DEBUG { 00:49:05 DEBUG "metric": { 00:49:05 DEBUG "pod": "ds-idrepo-1" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG [...] 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG "pod": "ds-idrepo-0" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 1713138545, 00:49:05 DEBUG "0" 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG } 00:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138545" 00:49:05 INFO [http_cmd]: http status code OK 00:49:05 DEBUG --- status code --- 00:49:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:05 DEBUG --- http response --- 00:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:49:05 DEBUG ----- output ----- 00:49:05 DEBUG { 00:49:05 DEBUG "status": "success", 00:49:05 DEBUG "data": { 00:49:05 DEBUG "resultType": "vector", 00:49:05 DEBUG "result": [ 00:49:05 DEBUG { 00:49:05 DEBUG "metric": { 00:49:05 DEBUG "pod": "ds-idrepo-1" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG [...] 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG "pod": "ds-cts-1" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 1713138545, 00:49:05 DEBUG "0.0001" 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG } 00:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138545" 00:49:05 INFO [http_cmd]: http status code OK 00:49:05 DEBUG --- status code --- 00:49:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:05 DEBUG --- http response --- 00:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:05 DEBUG ----- output ----- 00:49:05 DEBUG { 00:49:05 DEBUG "status": "success", 00:49:05 DEBUG "data": { 00:49:05 DEBUG "resultType": "vector", 00:49:05 DEBUG "result": [ 00:49:05 DEBUG { 00:49:05 DEBUG "metric": { 00:49:05 DEBUG "instance": "10.142.0.107:9100" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG [...] 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG "instance": "10.142.0.43:9100" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 1713138545, 00:49:05 DEBUG "5509.366666666667" 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG } 00:49:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138545" 00:49:05 INFO [http_cmd]: http status code OK 00:49:05 DEBUG --- status code --- 00:49:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:05 DEBUG --- http response --- 00:49:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:05 DEBUG ----- output ----- 00:49:05 DEBUG { 00:49:05 DEBUG "status": "success", 00:49:05 DEBUG "data": { 00:49:05 DEBUG "resultType": "vector", 00:49:05 DEBUG "result": [ 00:49:05 DEBUG { 00:49:05 DEBUG "metric": { 00:49:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG [...] 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG 00:49:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:49:05 DEBUG }, 00:49:05 DEBUG "value": [ 00:49:05 DEBUG 1713138545, 00:49:05 DEBUG "2290.9333333333334" 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG ] 00:49:05 DEBUG } 00:49:05 DEBUG } 00:49:21 INFO 00:49:21 INFO [loop_until]: kubectl --namespace=xlou top pods 00:49:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:49:22 INFO [loop_until]: OK (rc = 0) 00:49:22 DEBUG --- stdout --- 00:49:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12718m 8629Mi am-7d597cf7d6-8vr8f 11722m 8935Mi am-7d597cf7d6-9wz9h 16000m 8821Mi ds-cts-0 11780m 8185Mi ds-cts-1 12502m 8476Mi ds-cts-2 11758m 8214Mi ds-idrepo-0 13583m 4577Mi ds-idrepo-1 1947m 4303Mi ds-idrepo-2 1948m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6373m 5795Mi 00:49:22 DEBUG --- stderr --- 00:49:22 DEBUG 00:49:26 INFO 00:49:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:49:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:49:26 INFO [loop_until]: OK (rc = 0) 00:49:26 DEBUG --- stdout --- 00:49:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4880Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12173m 40% 11101Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13056m 43% 10699Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17517m 58% 3034Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2253m 7% 6823Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11658m 39% 12397Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2934Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12500m 41% 10415Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2004m 6% 5973Mi 5% gke-xlou-cdm-ds-0f601751-j488 12091m 40% 10726Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2247Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17074m 57% 10426Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13358m 44% 6502Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6547m 41% 9545Mi 16% 00:49:26 DEBUG --- stderr --- 00:49:26 DEBUG 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG { 00:50:04 DEBUG { 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG { 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG --- http response --- 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG "data": { 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG "data": { 00:50:04 DEBUG "data": { 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG "result": [ 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG "result": [ 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG { 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG { 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG { 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:04 DEBUG "metric": { 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "data": { 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG "pod": "ds-idrepo-1" 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG "pod": "ds-cts-2" 00:50:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:04 DEBUG "result": [ 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG { 00:50:04 DEBUG }, 00:50:04 DEBUG { 00:50:04 DEBUG }, 00:50:04 DEBUG }, 00:50:04 DEBUG { 00:50:04 DEBUG "result": [ 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "metric": { 00:50:04 DEBUG { 00:50:04 DEBUG "data": { 00:50:04 DEBUG 00:50:04 DEBUG "data": { 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "ds-idrepo-1" 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG }, 00:50:04 DEBUG "instance": "10.142.0.107:9100" 00:50:04 DEBUG "result": [ 00:50:04 DEBUG 00:50:04 DEBUG "result": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "value": [ 00:50:04 DEBUG }, 00:50:04 DEBUG { 00:50:04 DEBUG [...] 00:50:04 DEBUG { 00:50:04 DEBUG [...] 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "metric": { 00:50:04 DEBUG 00:50:04 DEBUG "metric": { 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "ds-idrepo-1" 00:50:04 DEBUG 00:50:04 DEBUG "instance": "10.142.0.107:9100" 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 ERROR [http_cmd]: ERROR 00:50:04 DEBUG }, 00:50:04 DEBUG 00:50:04 DEBUG }, 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG --- status code --- 00:50:04 ERROR --- status code --- 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "pod": "ds-idrepo-0" 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:50:04 DEBUG 00:50:04 DEBUG }, 00:50:04 DEBUG 00:50:04 DEBUG }, 00:50:04 DEBUG }, 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG --- http response --- 00:50:04 ERROR --- http response --- 00:50:04 DEBUG 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:04 DEBUG 00:50:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "0" 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG 00:50:04 DEBUG ----- output ----- 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG [...] 00:50:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:50:04 DEBUG "0" 00:50:04 DEBUG [...] 00:50:04 DEBUG "pod": "ds-idrepo-0" 00:50:04 DEBUG ] 00:50:04 DEBUG "2297.7" 00:50:04 DEBUG "instance": "10.142.0.43:9100" 00:50:04 DEBUG { 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG { 00:50:04 DEBUG 00:50:04 DEBUG ] 00:50:04 DEBUG 00:50:04 DEBUG }, 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG }, 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG 00:50:04 DEBUG } 00:50:04 DEBUG 00:50:04 DEBUG "value": [ 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "data": { 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG "data": { 00:50:04 DEBUG 00:50:04 DEBUG ] 00:50:04 DEBUG 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 DEBUG "pod": "ds-idrepo-0" 00:50:04 DEBUG } 00:50:04 DEBUG "instance": "10.142.0.43:9100" 00:50:04 DEBUG "0" 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG "0.00030000000000001136" 00:50:04 DEBUG "result": [ 00:50:04 DEBUG "result": [ 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG }, 00:50:04 DEBUG } 00:50:04 DEBUG }, 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG { 00:50:04 DEBUG { 00:50:04 DEBUG { 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "value": [ 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:04 DEBUG "pod": "ds-idrepo-1" 00:50:04 DEBUG "data": { 00:50:04 DEBUG "0" 00:50:04 DEBUG "13384899584" 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG }, 00:50:04 DEBUG }, 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG ] 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "result": [ 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG { 00:50:04 DEBUG ] 00:50:04 DEBUG ] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "metric": { 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG [...] 00:50:04 DEBUG [...] 00:50:04 DEBUG }, 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG "pod": "ds-cts-1" 00:50:04 DEBUG 00:50:04 DEBUG }, 00:50:04 DEBUG }, 00:50:04 DEBUG [...] 00:50:04 DEBUG "value": [ 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG 00:50:04 DEBUG "412979" 00:50:04 DEBUG "0" 00:50:04 DEBUG 00:50:04 DEBUG ] 00:50:04 DEBUG ] 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG }, 00:50:04 DEBUG ] 00:50:04 DEBUG ] 00:50:04 DEBUG "value": [ 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG "2297.2" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138604" 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "0.053" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "instance": "10.142.0.107:9100" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "instance": "10.142.0.43:9100" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "14393.833333333332" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "pod": "ds-cts-2" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "7.798555696569854" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG { 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "2280.8" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG { 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "0" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "6.409857448333438" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138604" 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "pod": "ds-cts-2" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "271005285.7263158" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138604" 00:50:04 INFO [http_cmd]: http status code OK 00:50:04 DEBUG --- status code --- 00:50:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:04 DEBUG --- http response --- 00:50:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:04 DEBUG ----- output ----- 00:50:04 DEBUG { 00:50:04 DEBUG "status": "success", 00:50:04 DEBUG "data": { 00:50:04 DEBUG "resultType": "vector", 00:50:04 DEBUG "result": [ 00:50:04 DEBUG { 00:50:04 DEBUG "metric": { 00:50:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG [...] 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG 00:50:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:50:04 DEBUG }, 00:50:04 DEBUG "value": [ 00:50:04 DEBUG 1713138604, 00:50:04 DEBUG "6178127872" 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG ] 00:50:04 DEBUG } 00:50:04 DEBUG } 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138605" 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138605" 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "pod": "ds-cts-2" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "4320146.670676691" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "pod": "ds-idrepo-1" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "pod": "ds-idrepo-0" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "0" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138605" 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "instance": "10.142.0.107:9100" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "instance": "10.142.0.43:9100" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "0" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138605" 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "pod": "ds-idrepo-1" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "pod": "ds-idrepo-0" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "0" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138605" 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "pod": "ds-idrepo-1" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "pod": "ds-cts-1" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "0.00030000000000000003" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138605" 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "instance": "10.142.0.107:9100" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "instance": "10.142.0.43:9100" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "5457.1" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138605" 00:50:05 INFO [http_cmd]: http status code OK 00:50:05 DEBUG --- status code --- 00:50:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:05 DEBUG --- http response --- 00:50:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:05 DEBUG ----- output ----- 00:50:05 DEBUG { 00:50:05 DEBUG "status": "success", 00:50:05 DEBUG "data": { 00:50:05 DEBUG "resultType": "vector", 00:50:05 DEBUG "result": [ 00:50:05 DEBUG { 00:50:05 DEBUG "metric": { 00:50:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG [...] 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG 00:50:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:50:05 DEBUG }, 00:50:05 DEBUG "value": [ 00:50:05 DEBUG 1713138605, 00:50:05 DEBUG "2299.1" 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG ] 00:50:05 DEBUG } 00:50:05 DEBUG } 00:50:22 INFO 00:50:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:50:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:50:22 INFO [loop_until]: OK (rc = 0) 00:50:22 DEBUG --- stdout --- 00:50:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12224m 8628Mi am-7d597cf7d6-8vr8f 11778m 8934Mi am-7d597cf7d6-9wz9h 16087m 8821Mi ds-cts-0 11604m 8190Mi ds-cts-1 11818m 8498Mi ds-cts-2 12038m 8201Mi ds-idrepo-0 13452m 4581Mi ds-idrepo-1 1963m 4303Mi ds-idrepo-2 1983m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6584m 5796Mi 00:50:22 DEBUG --- stderr --- 00:50:22 DEBUG 00:50:26 INFO 00:50:26 INFO [loop_until]: kubectl --namespace=xlou top node 00:50:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:50:26 INFO [loop_until]: OK (rc = 0) 00:50:27 DEBUG --- stdout --- 00:50:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4888Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11935m 39% 11090Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12888m 43% 10701Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17499m 58% 3048Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2215m 7% 6833Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11260m 37% 12378Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2936Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12566m 42% 10407Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2009m 6% 5969Mi 5% gke-xlou-cdm-ds-0f601751-j488 12073m 40% 10724Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2250Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16485m 55% 10416Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13411m 44% 6505Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6538m 41% 9544Mi 16% 00:50:27 DEBUG --- stderr --- 00:50:27 DEBUG 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138664" 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "pod": "ds-idrepo-1" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "ds-cts-1" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "0" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG --- http response --- 00:51:04 ERROR [http_cmd]: ERROR 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 ERROR --- status code --- 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG { 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 ERROR --- http response --- 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:51:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG "data": { 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG "status": "success", 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG "data": { 00:51:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG "data": { 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG "result": [ 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG "result": [ 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG { 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:04 DEBUG "data": { 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG }, 00:51:04 DEBUG { 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG { 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:04 DEBUG "result": [ 00:51:04 DEBUG "instance": "10.142.0.107:9100" 00:51:04 DEBUG }, 00:51:04 DEBUG { 00:51:04 DEBUG "value": [ 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG 00:51:04 DEBUG "data": { 00:51:04 DEBUG "data": { 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG "data": { 00:51:04 DEBUG 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "pod": "ds-idrepo-1" 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG 00:51:04 DEBUG "result": [ 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "data": { 00:51:04 DEBUG }, 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "result": [ 00:51:04 DEBUG [...] 00:51:04 DEBUG { 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG { 00:51:04 DEBUG 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "data": { 00:51:04 DEBUG "result": [ 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG "metric": { 00:51:04 DEBUG 00:51:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:51:04 DEBUG "pod": "ds-idrepo-1" 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG { 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "ds-idrepo-1" 00:51:04 DEBUG 00:51:04 DEBUG }, 00:51:04 DEBUG }, 00:51:04 DEBUG "result": [ 00:51:04 DEBUG "metric": { 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG }, 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG "value": [ 00:51:04 DEBUG "value": [ 00:51:04 DEBUG { 00:51:04 DEBUG "instance": "10.142.0.107:9100" 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG "instance": "10.142.0.43:9100" 00:51:04 DEBUG "value": [ 00:51:04 DEBUG }, 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "metric": { 00:51:04 DEBUG }, 00:51:04 DEBUG 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG }, 00:51:04 DEBUG 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG }, 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG 00:51:04 DEBUG "368557.5" 00:51:04 DEBUG [...] 00:51:04 DEBUG [...] 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG "pod": "ds-idrepo-0" 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "0.00026666666666651885" 00:51:04 DEBUG [...] 00:51:04 DEBUG ] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG }, 00:51:04 DEBUG "0" 00:51:04 DEBUG ] 00:51:04 DEBUG 00:51:04 DEBUG } 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG "value": [ 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG 00:51:04 DEBUG ] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:51:04 DEBUG "pod": "ds-idrepo-0" 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG "0" 00:51:04 DEBUG ] 00:51:04 DEBUG 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG }, 00:51:04 DEBUG }, 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG "pod": "ds-idrepo-0" 00:51:04 DEBUG "value": [ 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG "instance": "10.142.0.43:9100" 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG }, 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG 00:51:04 DEBUG }, 00:51:04 DEBUG ] 00:51:04 DEBUG "value": [ 00:51:04 DEBUG "0" 00:51:04 DEBUG "0" 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG "value": [ 00:51:04 DEBUG } 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG ] 00:51:04 DEBUG ] 00:51:04 DEBUG }, 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG } 00:51:04 DEBUG "0" 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG "value": [ 00:51:04 DEBUG "13385434112" 00:51:04 DEBUG ] 00:51:04 DEBUG ] 00:51:04 DEBUG ] 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG "0" 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "2308.366666666667" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "0.053" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG { 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "0" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "instance": "10.142.0.107:9100" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "instance": "10.142.0.43:9100" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "14726.033333333331" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138664" 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "5289725952" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "0" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: http status code OK 00:51:04 DEBUG --- status code --- 00:51:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:04 DEBUG --- http response --- 00:51:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:04 DEBUG ----- output ----- 00:51:04 DEBUG { 00:51:04 DEBUG "status": "success", 00:51:04 DEBUG "data": { 00:51:04 DEBUG "resultType": "vector", 00:51:04 DEBUG "result": [ 00:51:04 DEBUG { 00:51:04 DEBUG "metric": { 00:51:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG [...] 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG 00:51:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:51:04 DEBUG }, 00:51:04 DEBUG "value": [ 00:51:04 DEBUG 1713138664, 00:51:04 DEBUG "5.129254401396659" 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG ] 00:51:04 DEBUG } 00:51:04 DEBUG } 00:51:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138664" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138664, 00:51:05 DEBUG "0" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138665" 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138665" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "372.91736214378926" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "pod": "ds-idrepo-1" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "pod": "ds-idrepo-0" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "0" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138665" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "instance": "10.142.0.107:9100" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "instance": "10.142.0.43:9100" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "0.00033333333333333365" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138665" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "pod": "ds-idrepo-1" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "pod": "ds-idrepo-0" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "0" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138665" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "pod": "ds-idrepo-1" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "pod": "ds-cts-1" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "0.00013333333333333334" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138665" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "instance": "10.142.0.107:9100" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "instance": "10.142.0.43:9100" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "5513" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138665" 00:51:05 INFO [http_cmd]: http status code OK 00:51:05 DEBUG --- status code --- 00:51:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:05 DEBUG --- http response --- 00:51:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:05 DEBUG ----- output ----- 00:51:05 DEBUG { 00:51:05 DEBUG "status": "success", 00:51:05 DEBUG "data": { 00:51:05 DEBUG "resultType": "vector", 00:51:05 DEBUG "result": [ 00:51:05 DEBUG { 00:51:05 DEBUG "metric": { 00:51:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG [...] 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG 00:51:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:51:05 DEBUG }, 00:51:05 DEBUG "value": [ 00:51:05 DEBUG 1713138665, 00:51:05 DEBUG "0" 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG ] 00:51:05 DEBUG } 00:51:05 DEBUG } 00:51:22 INFO 00:51:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:51:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:51:22 INFO [loop_until]: OK (rc = 0) 00:51:22 DEBUG --- stdout --- 00:51:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 3384m 8623Mi am-7d597cf7d6-8vr8f 2012m 8944Mi am-7d597cf7d6-9wz9h 2653m 8819Mi ds-cts-0 5305m 8186Mi ds-cts-1 5456m 8488Mi ds-cts-2 5555m 8193Mi ds-idrepo-0 1302m 4580Mi ds-idrepo-1 164m 4303Mi ds-idrepo-2 168m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 4952Mi 00:51:22 DEBUG --- stderr --- 00:51:22 DEBUG 00:51:27 INFO 00:51:27 INFO [loop_until]: kubectl --namespace=xlou top node 00:51:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:51:27 INFO [loop_until]: OK (rc = 0) 00:51:27 DEBUG --- stdout --- 00:51:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4878Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5660m 18% 11080Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2260m 7% 10763Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3048Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 220m 0% 6833Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5538m 18% 12408Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2941Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2267m 7% 10456Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 228m 0% 5976Mi 5% gke-xlou-cdm-ds-0f601751-j488 5654m 18% 10731Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2241Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 2928m 9% 10482Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1378m 4% 6512Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 65m 0% 8709Mi 14% 00:51:27 DEBUG --- stderr --- 00:51:27 DEBUG 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138724" 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 DEBUG --- status code --- 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG --- http response --- 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG { 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG "data": { 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "result": [ 00:52:04 DEBUG { 00:52:04 DEBUG "metric": { 00:52:04 DEBUG "pod": "ds-idrepo-1" 00:52:04 DEBUG }, 00:52:04 DEBUG "value": [ 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG [...] 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG "pod": "ds-cts-1" 00:52:04 DEBUG }, 00:52:04 DEBUG "value": [ 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG "0" 00:52:04 DEBUG ] 00:52:04 DEBUG } 00:52:04 DEBUG ] 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG { 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG "data": { 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "result": [ 00:52:04 DEBUG { 00:52:04 DEBUG "metric": { 00:52:04 DEBUG "instance": "10.142.0.107:9100" 00:52:04 DEBUG }, 00:52:04 ERROR [http_cmd]: ERROR 00:52:04 DEBUG "value": [ 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 ERROR --- status code --- 00:52:04 DEBUG 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG --- status code --- 00:52:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:52:04 DEBUG 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 ERROR --- http response --- 00:52:04 DEBUG 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 DEBUG --- http response --- 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG [...] 00:52:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG { 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG 00:52:04 DEBUG { 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:52:04 DEBUG "instance": "10.142.0.43:9100" 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG "data": { 00:52:04 DEBUG }, 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG "data": { 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "value": [ 00:52:04 DEBUG { 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "result": [ 00:52:04 DEBUG 1713138724, 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG "result": [ 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 DEBUG { 00:52:04 DEBUG "0.0003333333333330302" 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG "data": { 00:52:04 DEBUG { 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG "metric": { 00:52:04 DEBUG ] 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "metric": { 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:04 DEBUG } 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG "result": [ 00:52:04 DEBUG "pod": "ds-idrepo-1" 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG }, 00:52:04 DEBUG ] 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:04 DEBUG { 00:52:04 DEBUG }, 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:04 DEBUG "value": [ 00:52:04 DEBUG } 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG "metric": { 00:52:04 DEBUG "value": [ 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG 00:52:04 DEBUG } 00:52:04 DEBUG { 00:52:04 DEBUG "pod": "ds-idrepo-1" 00:52:04 DEBUG 00:52:04 DEBUG { 00:52:04 DEBUG 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG }, 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 INFO [http_cmd]: http status code OK 00:52:04 DEBUG 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG 00:52:04 DEBUG "data": { 00:52:04 DEBUG "value": [ 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG --- status code --- 00:52:04 DEBUG 00:52:04 DEBUG "data": { 00:52:04 DEBUG [...] 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:04 DEBUG [...] 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG 00:52:04 DEBUG "result": [ 00:52:04 DEBUG 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG --- http response --- 00:52:04 DEBUG 00:52:04 DEBUG "result": [ 00:52:04 DEBUG 00:52:04 DEBUG { 00:52:04 DEBUG 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:04 DEBUG 00:52:04 DEBUG { 00:52:04 DEBUG 00:52:04 DEBUG "metric": { 00:52:04 DEBUG [...] 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG ----- output ----- 00:52:04 DEBUG 00:52:04 DEBUG "metric": { 00:52:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:04 DEBUG "pod": "ds-idrepo-1" 00:52:04 DEBUG 00:52:04 DEBUG { 00:52:04 DEBUG { 00:52:04 DEBUG { 00:52:04 DEBUG "pod": "ds-idrepo-0" 00:52:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:52:04 DEBUG }, 00:52:04 DEBUG }, 00:52:04 DEBUG 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG "status": "success", 00:52:04 DEBUG }, 00:52:04 DEBUG }, 00:52:04 DEBUG "value": [ 00:52:04 DEBUG "value": [ 00:52:04 DEBUG 00:52:04 DEBUG "data": { 00:52:04 DEBUG "data": { 00:52:04 DEBUG "data": { 00:52:04 DEBUG "value": [ 00:52:04 DEBUG "value": [ 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG 00:52:04 DEBUG "pod": "ds-idrepo-0" 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG "resultType": "vector", 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG 00:52:04 DEBUG "0" 00:52:04 DEBUG 00:52:04 DEBUG }, 00:52:04 DEBUG "result": [ 00:52:04 DEBUG "result": [ 00:52:04 DEBUG "result": [ 00:52:04 DEBUG "0" 00:52:04 DEBUG 00:52:04 DEBUG ] 00:52:04 DEBUG 00:52:04 DEBUG "value": [ 00:52:04 DEBUG { 00:52:04 DEBUG { 00:52:04 DEBUG { 00:52:04 DEBUG ] 00:52:04 DEBUG 00:52:04 DEBUG } 00:52:04 DEBUG [...] 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG "metric": { 00:52:04 DEBUG "metric": { 00:52:04 DEBUG "metric": { 00:52:04 DEBUG } 00:52:04 DEBUG [...] 00:52:04 DEBUG ] 00:52:04 DEBUG 00:52:04 DEBUG "0" 00:52:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:04 DEBUG "instance": "10.142.0.107:9100" 00:52:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:04 DEBUG ] 00:52:04 DEBUG 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG ] 00:52:04 DEBUG }, 00:52:04 DEBUG }, 00:52:04 DEBUG }, 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG } 00:52:04 DEBUG "value": [ 00:52:04 DEBUG "value": [ 00:52:04 DEBUG "value": [ 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG "pod": "ds-idrepo-0" 00:52:04 DEBUG ] 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:52:04 DEBUG }, 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG }, 00:52:04 DEBUG "value": [ 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG "value": [ 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG [...] 00:52:04 DEBUG [...] 00:52:04 DEBUG [...] 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG "0" 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG "0" 00:52:04 DEBUG ] 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG ] 00:52:04 DEBUG } 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG 00:52:04 DEBUG } 00:52:04 DEBUG ] 00:52:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:04 DEBUG "instance": "10.142.0.43:9100" 00:52:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:04 DEBUG ] 00:52:04 DEBUG } 00:52:04 DEBUG }, 00:52:04 DEBUG }, 00:52:04 DEBUG }, 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG "value": [ 00:52:04 DEBUG "value": [ 00:52:04 DEBUG "value": [ 00:52:04 DEBUG } 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG 1713138724, 00:52:04 DEBUG "0" 00:52:04 DEBUG "13386128384" 00:52:04 DEBUG "230485.5" 00:52:04 DEBUG ] 00:52:04 DEBUG ] 00:52:04 DEBUG ] 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG ] 00:52:04 DEBUG ] 00:52:04 DEBUG ] 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 DEBUG } 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138724" 00:52:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138724" 00:52:04 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138724, 00:52:05 DEBUG "2297.2666666666664" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138724, 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138725" 00:52:05 DEBUG "5289713664" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138724, 00:52:05 DEBUG "0.026780415374785697" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 DEBUG ----- output ----- 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG { 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG "data": { 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG }, 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG "value": [ 00:52:05 DEBUG { 00:52:05 DEBUG 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG 00:52:05 DEBUG "data": { 00:52:05 DEBUG 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG [...] 00:52:05 DEBUG "result": [ 00:52:05 DEBUG 00:52:05 DEBUG { 00:52:05 DEBUG 00:52:05 DEBUG "metric": { 00:52:05 DEBUG 00:52:05 DEBUG "instance": "10.142.0.107:9100" 00:52:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:05 DEBUG }, 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG 00:52:05 DEBUG "0.053" 00:52:05 DEBUG 00:52:05 DEBUG ] 00:52:05 DEBUG [...] 00:52:05 DEBUG } 00:52:05 DEBUG 00:52:05 DEBUG ] 00:52:05 DEBUG 00:52:05 DEBUG } 00:52:05 DEBUG 00:52:05 DEBUG } 00:52:05 DEBUG "instance": "10.142.0.43:9100" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "14756.133333333331" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0.0034166692015210193" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "4609.946768060837" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "ds-idrepo-1" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "ds-idrepo-0" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "373.7794676806084" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "ds-idrepo-1" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "ds-idrepo-0" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "instance": "10.142.0.107:9100" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "instance": "10.142.0.43:9100" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0.00033333333333333365" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "ds-idrepo-1" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "ds-cts-1" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0.0002" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "instance": "10.142.0.107:9100" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "instance": "10.142.0.43:9100" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "5658.9" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138725" 00:52:05 INFO [http_cmd]: http status code OK 00:52:05 DEBUG --- status code --- 00:52:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:05 DEBUG --- http response --- 00:52:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:05 DEBUG ----- output ----- 00:52:05 DEBUG { 00:52:05 DEBUG "status": "success", 00:52:05 DEBUG "data": { 00:52:05 DEBUG "resultType": "vector", 00:52:05 DEBUG "result": [ 00:52:05 DEBUG { 00:52:05 DEBUG "metric": { 00:52:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG [...] 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG 00:52:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:52:05 DEBUG }, 00:52:05 DEBUG "value": [ 00:52:05 DEBUG 1713138725, 00:52:05 DEBUG "0" 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG ] 00:52:05 DEBUG } 00:52:05 DEBUG } 00:52:18 WARNING File /tmp/lodemon_data-20240414_182729/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 00:52:18] "GET /monitoring/average?start_time=20240414_225133&stop_time=20240414_235017 HTTP/1.1" 200 - 00:52:22 INFO 00:52:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:52:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:52:22 INFO [loop_until]: OK (rc = 0) 00:52:22 DEBUG --- stdout --- 00:52:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2149m 8549Mi am-7d597cf7d6-8vr8f 1997m 8908Mi am-7d597cf7d6-9wz9h 2604m 8744Mi ds-cts-0 5363m 8240Mi ds-cts-1 5626m 8527Mi ds-cts-2 5516m 8153Mi ds-idrepo-0 1255m 4581Mi ds-idrepo-1 213m 4303Mi ds-idrepo-2 169m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 4m 68Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 4952Mi 00:52:22 DEBUG --- stderr --- 00:52:22 DEBUG 00:52:27 INFO 00:52:27 INFO [loop_until]: kubectl --namespace=xlou top node 00:52:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:52:27 INFO [loop_until]: OK (rc = 0) 00:52:27 DEBUG --- stdout --- 00:52:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4876Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5593m 18% 11091Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2197m 7% 10667Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3025Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 219m 0% 6828Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5229m 17% 12426Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2937Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2146m 7% 10404Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 233m 0% 5985Mi 5% gke-xlou-cdm-ds-0f601751-j488 5645m 18% 10726Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 2240Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 2888m 9% 10375Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1325m 4% 6506Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 484m 3% 8967Mi 15% 00:52:27 DEBUG --- stderr --- 00:52:27 DEBUG 00:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138784" 00:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138784" 00:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138784" 00:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138784" 00:53:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138784" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "data": { 00:53:05 DEBUG "data": { 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG "result": [ 00:53:05 DEBUG "result": [ 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG "instance": "10.142.0.107:9100" 00:53:05 DEBUG "instance": "10.142.0.107:9100" 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG [...] 00:53:05 DEBUG [...] 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG "instance": "10.142.0.43:9100" 00:53:05 DEBUG "instance": "10.142.0.43:9100" 00:53:05 DEBUG "pod": "ds-idrepo-0" 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138785" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG 1713138784, 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG 1713138784, 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG 1713138784, 00:53:05 DEBUG 1713138784, 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG "501.06666666666666" 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG "0.00036666666666652263" 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG "13386763264" 00:53:05 DEBUG "0" 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG ] 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG ] 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG } 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG } 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:05 DEBUG ] 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ] 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:53:05 DEBUG ] 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138785" 00:53:05 DEBUG ] 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG } 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG } 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG } 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG } 00:53:05 DEBUG { 00:53:05 DEBUG } 00:53:05 DEBUG { 00:53:05 DEBUG } 00:53:05 DEBUG { 00:53:05 DEBUG } 00:53:05 DEBUG { 00:53:05 DEBUG } 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "data": { 00:53:05 DEBUG "data": { 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG "result": [ 00:53:05 DEBUG "result": [ 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG [...] 00:53:05 DEBUG [...] 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "ds-cts-1" 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:53:05 DEBUG "pod": "ds-idrepo-0" 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0" 00:53:05 DEBUG "100675.5" 00:53:05 DEBUG "0" 00:53:05 DEBUG "0" 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "5708054528" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 ERROR [http_cmd]: ERROR 00:53:05 ERROR --- status code --- 00:53:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:53:05 ERROR --- http response --- 00:53:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:53:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "501.46666666666664" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0.5575937732960116" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "ds-idrepo-0" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "2296.8333333333335" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0.053" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "instance": "10.142.0.107:9100" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "instance": "10.142.0.43:9100" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "14999.766666666665" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0.003202804302492478" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "6181.538799253649" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "224.78322906376906" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "ds-idrepo-0" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "ds-idrepo-0" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "instance": "10.142.0.107:9100" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "instance": "10.142.0.43:9100" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0.00033333333333333365" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "ds-idrepo-1" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "ds-cts-1" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "0.00006666666666666667" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "instance": "10.142.0.107:9100" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "instance": "10.142.0.43:9100" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "5808.133333333333" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138785" 00:53:05 INFO [http_cmd]: http status code OK 00:53:05 DEBUG --- status code --- 00:53:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:05 DEBUG --- http response --- 00:53:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:05 DEBUG ----- output ----- 00:53:05 DEBUG { 00:53:05 DEBUG "status": "success", 00:53:05 DEBUG "data": { 00:53:05 DEBUG "resultType": "vector", 00:53:05 DEBUG "result": [ 00:53:05 DEBUG { 00:53:05 DEBUG "metric": { 00:53:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG [...] 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG 00:53:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:53:05 DEBUG }, 00:53:05 DEBUG "value": [ 00:53:05 DEBUG 1713138785, 00:53:05 DEBUG "501.53333333333336" 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG ] 00:53:05 DEBUG } 00:53:05 DEBUG } 00:53:22 INFO 00:53:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:53:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:53:22 INFO [loop_until]: OK (rc = 0) 00:53:22 DEBUG --- stdout --- 00:53:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 3724m 8553Mi am-7d597cf7d6-8vr8f 3579m 8928Mi am-7d597cf7d6-9wz9h 4705m 8747Mi ds-cts-0 6525m 8154Mi ds-cts-1 7002m 8468Mi ds-cts-2 6523m 8160Mi ds-idrepo-0 4385m 4581Mi ds-idrepo-1 504m 4302Mi ds-idrepo-2 544m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 9m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 2093m 5360Mi 00:53:22 DEBUG --- stderr --- 00:53:22 DEBUG 00:53:27 INFO 00:53:27 INFO [loop_until]: kubectl --namespace=xlou top node 00:53:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:53:27 INFO [loop_until]: OK (rc = 0) 00:53:27 DEBUG --- stdout --- 00:53:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4865Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10301m 34% 11091Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12306m 41% 10631Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 19056m 63% 3084Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2118m 7% 6830Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9540m 31% 12434Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2939Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12162m 40% 10403Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2054m 6% 5974Mi 5% gke-xlou-cdm-ds-0f601751-j488 9205m 30% 10751Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 2231Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 17065m 57% 10416Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13735m 46% 6514Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7299m 45% 9537Mi 16% 00:53:27 DEBUG --- stderr --- 00:53:27 DEBUG 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "instance": "10.142.0.107:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "instance": "10.142.0.43:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "13387418624" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG { 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG "data": { 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG "result": [ 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG { 00:54:05 DEBUG { 00:54:05 DEBUG { 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "metric": { 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "data": { 00:54:05 DEBUG "data": { 00:54:05 DEBUG "instance": "10.142.0.107:9100" 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG }, 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG "result": [ 00:54:05 DEBUG "result": [ 00:54:05 DEBUG "value": [ 00:54:05 DEBUG "result": [ 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG { 00:54:05 DEBUG { 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG { 00:54:05 DEBUG 00:54:05 DEBUG { 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG "metric": { 00:54:05 DEBUG 00:54:05 DEBUG "metric": { 00:54:05 DEBUG { 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG }, 00:54:05 DEBUG }, 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG }, 00:54:05 DEBUG [...] 00:54:05 DEBUG }, 00:54:05 DEBUG "data": { 00:54:05 DEBUG "value": [ 00:54:05 DEBUG "value": [ 00:54:05 DEBUG "value": [ 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG 00:54:05 DEBUG "value": [ 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "result": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG { 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG { 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "instance": "10.142.0.43:9100" 00:54:05 DEBUG 00:54:05 DEBUG "metric": { 00:54:05 DEBUG [...] 00:54:05 DEBUG [...] 00:54:05 DEBUG [...] 00:54:05 DEBUG "data": { 00:54:05 DEBUG }, 00:54:05 DEBUG [...] 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG }, 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG "result": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG { 00:54:05 DEBUG "0.00033333333333350387" 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "ds-idrepo-0" 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG "pod": "ds-cts-1" 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG "metric": { 00:54:05 DEBUG ] 00:54:05 DEBUG "pod": "ds-idrepo-0" 00:54:05 DEBUG 00:54:05 DEBUG }, 00:54:05 DEBUG }, 00:54:05 DEBUG }, 00:54:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:54:05 DEBUG } 00:54:05 DEBUG }, 00:54:05 DEBUG 00:54:05 DEBUG "value": [ 00:54:05 DEBUG "value": [ 00:54:05 DEBUG "value": [ 00:54:05 DEBUG }, 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG ] 00:54:05 DEBUG "value": [ 00:54:05 DEBUG [...] 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG { 00:54:05 DEBUG } 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG 00:54:05 DEBUG "0" 00:54:05 DEBUG "2475.8" 00:54:05 DEBUG "0" 00:54:05 DEBUG 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG } 00:54:05 DEBUG "0" 00:54:05 DEBUG 00:54:05 DEBUG ] 00:54:05 DEBUG ] 00:54:05 DEBUG ] 00:54:05 DEBUG 00:54:05 DEBUG "data": { 00:54:05 DEBUG ] 00:54:05 DEBUG 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG } 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG ] 00:54:05 DEBUG ] 00:54:05 DEBUG ] 00:54:05 DEBUG [...] 00:54:05 DEBUG "result": [ 00:54:05 DEBUG ] 00:54:05 DEBUG }, 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG { 00:54:05 DEBUG } 00:54:05 DEBUG "value": [ 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG "metric": { 00:54:05 DEBUG } 00:54:05 DEBUG 1713138845, 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 DEBUG 00:54:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:54:05 DEBUG "122011.5" 00:54:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:54:05 DEBUG }, 00:54:05 DEBUG ] 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG } 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG ] 00:54:05 DEBUG 00:54:05 DEBUG "6173782016" 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG } 00:54:05 DEBUG [...] 00:54:05 DEBUG ] 00:54:05 DEBUG 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG } 00:54:05 DEBUG 00:54:05 DEBUG "pod": "lodemon-5d8fc565dd-9g64c" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "ds-idrepo-0" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "2476.0666666666666" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138845" 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "6.604710329056377" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 ERROR [http_cmd]: ERROR 00:54:05 ERROR --- status code --- 00:54:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:54:05 ERROR --- http response --- 00:54:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:54:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138845" 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0.054" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "instance": "10.142.0.107:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "instance": "10.142.0.43:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "14760.033333333333" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "ds-idrepo-0" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "lodemon-5d8fc565dd-9g64c" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "3583.2385618056164" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "ds-idrepo-0" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "instance": "10.142.0.107:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "instance": "10.142.0.43:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "ds-idrepo-1" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "ds-cts-1" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "0.000033333333333333335" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "instance": "10.142.0.107:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "instance": "10.142.0.43:9100" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "5625.033333333333" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138845" 00:54:05 INFO [http_cmd]: http status code OK 00:54:05 DEBUG --- status code --- 00:54:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:05 DEBUG --- http response --- 00:54:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:05 DEBUG ----- output ----- 00:54:05 DEBUG { 00:54:05 DEBUG "status": "success", 00:54:05 DEBUG "data": { 00:54:05 DEBUG "resultType": "vector", 00:54:05 DEBUG "result": [ 00:54:05 DEBUG { 00:54:05 DEBUG "metric": { 00:54:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG [...] 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG 00:54:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:54:05 DEBUG }, 00:54:05 DEBUG "value": [ 00:54:05 DEBUG 1713138845, 00:54:05 DEBUG "2475.8333333333335" 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG ] 00:54:05 DEBUG } 00:54:05 DEBUG } 00:54:22 INFO 00:54:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:54:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:54:22 INFO [loop_until]: OK (rc = 0) 00:54:22 DEBUG --- stdout --- 00:54:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 10915m 8572Mi am-7d597cf7d6-8vr8f 10345m 8914Mi am-7d597cf7d6-9wz9h 14030m 8818Mi ds-cts-0 5386m 8186Mi ds-cts-1 5438m 8475Mi ds-cts-2 5349m 8159Mi ds-idrepo-0 13049m 4581Mi ds-idrepo-1 2043m 4303Mi ds-idrepo-2 1901m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7050m 5792Mi 00:54:22 DEBUG --- stderr --- 00:54:22 DEBUG 00:54:27 INFO 00:54:27 INFO [loop_until]: kubectl --namespace=xlou top node 00:54:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:54:27 INFO [loop_until]: OK (rc = 0) 00:54:27 DEBUG --- stdout --- 00:54:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4871Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5372m 17% 11134Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11239m 37% 10610Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 19365m 64% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2227m 7% 6819Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5249m 17% 12480Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10757m 36% 10150Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1956m 6% 5980Mi 5% gke-xlou-cdm-ds-0f601751-j488 5451m 18% 10777Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2251Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 14724m 49% 10397Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12823m 42% 6512Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7123m 44% 9539Mi 16% 00:54:27 DEBUG --- stderr --- 00:54:27 DEBUG 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- status code --- 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG --- status code --- 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 ERROR [http_cmd]: ERROR 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- status code --- 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 ERROR --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- status code --- 00:55:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 ERROR --- http response --- 00:55:05 DEBUG { 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG { 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG { 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG { 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "status": "success", 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:55:05 DEBUG { 00:55:05 DEBUG "data": { 00:55:05 DEBUG { 00:55:05 DEBUG --- http response --- 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138905" 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "data": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "data": { 00:55:05 DEBUG { 00:55:05 DEBUG "data": { 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:55:05 DEBUG { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "data": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "data": { 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "instance": "10.142.0.107:9100" 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "data": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG { 00:55:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:55:05 DEBUG { 00:55:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:55:05 DEBUG }, 00:55:05 DEBUG { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG }, 00:55:05 DEBUG "metric": { 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "data": { 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG { 00:55:05 DEBUG "instance": "10.142.0.107:9100" 00:55:05 DEBUG { 00:55:05 DEBUG "data": { 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG { 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG }, 00:55:05 DEBUG "metric": { 00:55:05 DEBUG }, 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG "metric": { 00:55:05 DEBUG }, 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG "result": [ 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG "value": [ 00:55:05 DEBUG { 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG { 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG "metric": { 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "metric": { 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "ds-cts-2" 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [...] 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "instance": "10.142.0.43:9100" 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG [...] 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:55:05 DEBUG 00:55:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG { 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG 00:55:05 DEBUG "instance": "10.142.0.43:9100" 00:55:05 DEBUG }, 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "ds-idrepo-0" 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG [...] 00:55:05 DEBUG }, 00:55:05 DEBUG "data": { 00:55:05 DEBUG 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG }, 00:55:05 DEBUG "6169624576" 00:55:05 DEBUG "13387949056" 00:55:05 DEBUG ] 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "pod": "ds-idrepo-0" 00:55:05 DEBUG 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG { 00:55:05 DEBUG "6.974790990719136" 00:55:05 DEBUG "value": [ 00:55:05 DEBUG ] 00:55:05 DEBUG "pod": "ds-cts-1" 00:55:05 DEBUG } 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "result": [ 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0.00030000000000001136" 00:55:05 DEBUG "result": [ 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG ] 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG } 00:55:05 DEBUG }, 00:55:05 DEBUG ] 00:55:05 DEBUG }, 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG 00:55:05 DEBUG "0" 00:55:05 DEBUG { 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "pod": "ds-idrepo-0" 00:55:05 DEBUG 00:55:05 DEBUG ] 00:55:05 DEBUG { 00:55:05 DEBUG "data": { 00:55:05 DEBUG } 00:55:05 DEBUG "2482.0666666666666" 00:55:05 DEBUG ] 00:55:05 DEBUG "value": [ 00:55:05 DEBUG } 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "270769.5" 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG ] 00:55:05 DEBUG "metric": { 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG } 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG ] 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG } 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG }, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG ] 00:55:05 DEBUG "pod": "ds-cts-2" 00:55:05 DEBUG "result": [ 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG "0" 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG ] 00:55:05 DEBUG ] 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:55:05 DEBUG }, 00:55:05 DEBUG { 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG ] 00:55:05 DEBUG }, 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG } 00:55:05 DEBUG 00:55:05 DEBUG } 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG "metric": { 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG "value": [ 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG 00:55:05 DEBUG } 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG "instance": "10.142.0.107:9100" 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG [...] 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG 00:55:05 DEBUG }, 00:55:05 DEBUG } 00:55:05 DEBUG "2481.733333333333" 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG 00:55:05 DEBUG "0" 00:55:05 DEBUG 00:55:05 DEBUG "value": [ 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG 00:55:05 DEBUG ] 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG 00:55:05 DEBUG } 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG ] 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG ] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG } 00:55:05 DEBUG }, 00:55:05 DEBUG } 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG } 00:55:05 DEBUG "value": [ 00:55:05 DEBUG } 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG }, 00:55:05 DEBUG 00:55:05 DEBUG "0" 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG ] 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "instance": "10.142.0.43:9100" 00:55:05 DEBUG } 00:55:05 DEBUG "0" 00:55:05 DEBUG }, 00:55:05 DEBUG ] 00:55:05 DEBUG ] 00:55:05 DEBUG "value": [ 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG "14362.666666666664" 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0.054" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "ds-cts-2" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "ds-idrepo-0" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "ds-cts-2" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "ds-idrepo-0" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "instance": "10.142.0.107:9100" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "instance": "10.142.0.43:9100" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "ds-idrepo-1" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "ds-cts-1" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "0" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "instance": "10.142.0.107:9100" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "instance": "10.142.0.43:9100" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "5537.4" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138905" 00:55:05 INFO [http_cmd]: http status code OK 00:55:05 DEBUG --- status code --- 00:55:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:05 DEBUG --- http response --- 00:55:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:05 DEBUG ----- output ----- 00:55:05 DEBUG { 00:55:05 DEBUG "status": "success", 00:55:05 DEBUG "data": { 00:55:05 DEBUG "resultType": "vector", 00:55:05 DEBUG "result": [ 00:55:05 DEBUG { 00:55:05 DEBUG "metric": { 00:55:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG [...] 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG 00:55:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:55:05 DEBUG }, 00:55:05 DEBUG "value": [ 00:55:05 DEBUG 1713138905, 00:55:05 DEBUG "2482.1" 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG ] 00:55:05 DEBUG } 00:55:05 DEBUG } 00:55:22 INFO 00:55:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:55:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:55:22 INFO [loop_until]: OK (rc = 0) 00:55:22 DEBUG --- stdout --- 00:55:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 11051m 8560Mi am-7d597cf7d6-8vr8f 9905m 8674Mi am-7d597cf7d6-9wz9h 13987m 8800Mi ds-cts-0 5268m 8249Mi ds-cts-1 5558m 8515Mi ds-cts-2 5394m 8214Mi ds-idrepo-0 12672m 4581Mi ds-idrepo-1 2041m 4303Mi ds-idrepo-2 1983m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7019m 5792Mi 00:55:22 DEBUG --- stderr --- 00:55:22 DEBUG 00:55:27 INFO 00:55:27 INFO [loop_until]: kubectl --namespace=xlou top node 00:55:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:55:27 INFO [loop_until]: OK (rc = 0) 00:55:27 DEBUG --- stdout --- 00:55:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5677m 19% 11155Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11173m 37% 10614Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 18777m 62% 3046Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2221m 7% 6830Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5444m 18% 12496Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10635m 35% 10135Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1966m 6% 5978Mi 5% gke-xlou-cdm-ds-0f601751-j488 5705m 19% 10781Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2236Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 14810m 49% 10399Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12717m 42% 6497Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7171m 45% 9540Mi 16% 00:55:27 DEBUG --- stderr --- 00:55:27 DEBUG 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 ERROR [http_cmd]: ERROR 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138965" 00:56:05 DEBUG --- http response --- 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG --- status code --- 00:56:05 ERROR --- status code --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:56:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG ----- output ----- 00:56:05 ERROR --- http response --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG { 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG { 00:56:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:56:05 DEBUG { 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:05 DEBUG "status": "success", 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG "status": "success", 00:56:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG "data": { 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG { 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG { 00:56:05 DEBUG "data": { 00:56:05 DEBUG "data": { 00:56:05 DEBUG { 00:56:05 DEBUG { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG "data": { 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "status": "success", 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG "result": [ 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG { 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "data": { 00:56:05 DEBUG "data": { 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG { 00:56:05 DEBUG "data": { 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "data": { 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "result": [ 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG { 00:56:05 DEBUG "data": { 00:56:05 DEBUG "result": [ 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG { 00:56:05 DEBUG "instance": "10.142.0.107:9100" 00:56:05 DEBUG { 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG { 00:56:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:56:05 DEBUG "pod": "ds-cts-2" 00:56:05 DEBUG { 00:56:05 DEBUG { 00:56:05 DEBUG }, 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG "metric": { 00:56:05 DEBUG }, 00:56:05 DEBUG "metric": { 00:56:05 DEBUG }, 00:56:05 DEBUG }, 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "value": [ 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "data": { 00:56:05 DEBUG "result": [ 00:56:05 DEBUG }, 00:56:05 DEBUG "metric": { 00:56:05 DEBUG }, 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG { 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "pod": "ds-cts-2" 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG }, 00:56:05 DEBUG { 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG "result": [ 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "instance": "10.142.0.107:9100" 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "metric": { 00:56:05 DEBUG 00:56:05 DEBUG "data": { 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "instance": "10.142.0.107:9100" 00:56:05 DEBUG }, 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:56:05 DEBUG 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG "result": [ 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG [...] 00:56:05 DEBUG { 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "metric": { 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG "pod": "ds-idrepo-0" 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG "pod": "ds-cts-1" 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "instance": "10.142.0.43:9100" 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG [...] 00:56:05 DEBUG "pod": "ds-idrepo-0" 00:56:05 DEBUG "value": [ 00:56:05 DEBUG }, 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "412577" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "0" 00:56:05 DEBUG "instance": "10.142.0.43:9100" 00:56:05 DEBUG "value": [ 00:56:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG 00:56:05 DEBUG }, 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG [...] 00:56:05 DEBUG ] 00:56:05 DEBUG }, 00:56:05 DEBUG "0.00033333333333350387" 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG 00:56:05 DEBUG ] 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "6169526272" 00:56:05 DEBUG 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:56:05 DEBUG }, 00:56:05 DEBUG "pod": "ds-idrepo-0" 00:56:05 DEBUG } 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG }, 00:56:05 DEBUG "13388624896" 00:56:05 DEBUG ] 00:56:05 DEBUG 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG 00:56:05 DEBUG "0" 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG }, 00:56:05 DEBUG ] 00:56:05 DEBUG "498.56666666666666" 00:56:05 DEBUG "value": [ 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:56:05 DEBUG "0" 00:56:05 DEBUG [...] 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG "instance": "10.142.0.43:9100" 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG }, 00:56:05 DEBUG ] 00:56:05 DEBUG 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "2483.2" 00:56:05 DEBUG "0" 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG "value": [ 00:56:05 DEBUG } 00:56:05 DEBUG 00:56:05 DEBUG ] 00:56:05 DEBUG }, 00:56:05 DEBUG } 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG ] 00:56:05 DEBUG ] 00:56:05 DEBUG ] 00:56:05 DEBUG "0" 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG ] 00:56:05 DEBUG 00:56:05 DEBUG } 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0.000013807054440082247" 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG "7.013047903646964" 00:56:05 DEBUG } 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG "14387.666666666664" 00:56:05 DEBUG ] 00:56:05 DEBUG ] 00:56:05 DEBUG ] 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG }, 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG "value": [ 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 DEBUG "2482.7666666666664" 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0.054" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "ds-cts-2" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "ds-idrepo-0" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "ds-cts-2" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138965" 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "instance": "10.142.0.107:9100" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "instance": "10.142.0.43:9100" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "ds-idrepo-0" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "ds-idrepo-1" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "ds-cts-1" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "0.00006086111111111111" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "instance": "10.142.0.107:9100" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "instance": "10.142.0.43:9100" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "5560.766666666666" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713138965" 00:56:05 INFO [http_cmd]: http status code OK 00:56:05 DEBUG --- status code --- 00:56:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:05 DEBUG --- http response --- 00:56:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:05 DEBUG ----- output ----- 00:56:05 DEBUG { 00:56:05 DEBUG "status": "success", 00:56:05 DEBUG "data": { 00:56:05 DEBUG "resultType": "vector", 00:56:05 DEBUG "result": [ 00:56:05 DEBUG { 00:56:05 DEBUG "metric": { 00:56:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG [...] 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG 00:56:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:56:05 DEBUG }, 00:56:05 DEBUG "value": [ 00:56:05 DEBUG 1713138965, 00:56:05 DEBUG "2484.633333333333" 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG ] 00:56:05 DEBUG } 00:56:05 DEBUG } 00:56:22 INFO 00:56:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:56:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:56:22 INFO [loop_until]: OK (rc = 0) 00:56:22 DEBUG --- stdout --- 00:56:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12065m 8569Mi am-7d597cf7d6-8vr8f 11018m 8677Mi am-7d597cf7d6-9wz9h 14285m 8814Mi ds-cts-0 6596m 8253Mi ds-cts-1 6287m 8515Mi ds-cts-2 6980m 8220Mi ds-idrepo-0 13112m 4582Mi ds-idrepo-1 2013m 4303Mi ds-idrepo-2 1999m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 7m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7102m 5793Mi 00:56:22 DEBUG --- stderr --- 00:56:22 DEBUG 00:56:27 INFO 00:56:27 INFO [loop_until]: kubectl --namespace=xlou top node 00:56:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:56:27 INFO [loop_until]: OK (rc = 0) 00:56:27 DEBUG --- stdout --- 00:56:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 4874Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12327m 41% 11102Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14309m 47% 10742Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 20095m 67% 3231Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2384m 7% 6817Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 12014m 40% 12450Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 13417m 44% 10142Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2119m 7% 5979Mi 5% gke-xlou-cdm-ds-0f601751-j488 12961m 43% 10719Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2239Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18047m 60% 10519Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14288m 47% 6496Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7275m 45% 9777Mi 16% 00:56:27 DEBUG --- stderr --- 00:56:27 DEBUG 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 DEBUG "instance": "10.142.0.107:9100" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 ERROR [http_cmd]: ERROR 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG }, 00:57:05 DEBUG --- status code --- 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 ERROR --- status code --- 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG "value": [ 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG --- status code --- 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- status code --- 00:57:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 ERROR --- http response --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:57:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG [...] 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG { 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG { 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG { 00:57:05 DEBUG "data": { 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "instance": "10.142.0.43:9100" 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "data": { 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG { 00:57:05 DEBUG { 00:57:05 DEBUG "resultType": "vector", 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "metric": { 00:57:05 DEBUG }, 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG { 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "result": [ 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG { 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-cts-2" 00:57:05 DEBUG "metric": { 00:57:05 DEBUG { 00:57:05 DEBUG "data": { 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG { 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG "metric": { 00:57:05 DEBUG }, 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "instance": "10.142.0.107:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG }, 00:57:05 DEBUG "metric": { 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG "pod": "ds-cts-2" 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "13389272064" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG }, 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "instance": "10.142.0.107:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:05 DEBUG 00:57:05 DEBUG ] 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG }, 00:57:05 DEBUG { 00:57:05 DEBUG 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG 00:57:05 DEBUG } 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "metric": { 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG { 00:57:05 DEBUG 00:57:05 DEBUG ] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG [...] 00:57:05 DEBUG } 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG }, 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "data": { 00:57:05 DEBUG 00:57:05 DEBUG } 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG "result": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG "instance": "10.142.0.43:9100" 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG { 00:57:05 DEBUG "pod": "ds-idrepo-0" 00:57:05 DEBUG }, 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG }, 00:57:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:57:05 DEBUG 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "metric": { 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG }, 00:57:05 DEBUG [...] 00:57:05 DEBUG }, 00:57:05 DEBUG 00:57:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "instance": "10.142.0.43:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG }, 00:57:05 DEBUG "0" 00:57:05 DEBUG }, 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG "13809.599999999999" 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "value": [ 00:57:05 DEBUG ] 00:57:05 DEBUG "value": [ 00:57:05 DEBUG }, 00:57:05 DEBUG ] 00:57:05 DEBUG "0.000011915839699370822" 00:57:05 DEBUG 00:57:05 DEBUG "2470.4333333333334" 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG 00:57:05 DEBUG } 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG ] 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG 00:57:05 DEBUG ] 00:57:05 DEBUG "441550.5" 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG }, 00:57:05 DEBUG } 00:57:05 DEBUG "0.0005666666666665302" 00:57:05 DEBUG ] 00:57:05 DEBUG 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG "2469.9666666666667" 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG "value": [ 00:57:05 DEBUG ] 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG [...] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 DEBUG ] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "ds-cts-1" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG }, 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG "0" 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139025" 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139025" 00:57:05 DEBUG ] 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG } 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG } 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG[0m "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "ds-idrepo-0" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "ds-idrepo-0" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG ----- output ----- 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "data": { 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG }, 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG "result": [ 00:57:05 DEBUG "value": [ 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:57:05 DEBUG 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 DEBUG "data": { 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG "result": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG { 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "metric": { 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG 00:57:05 DEBUG }, 00:57:05 DEBUG }, 00:57:05 DEBUG 00:57:05 DEBUG "value": [ 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:57:05 DEBUG 00:57:05 DEBUG "2602.2" 00:57:05 DEBUG }, 00:57:05 DEBUG 00:57:05 DEBUG ] 00:57:05 DEBUG "value": [ 00:57:05 DEBUG [...] 00:57:05 DEBUG } 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG 00:57:05 DEBUG ] 00:57:05 DEBUG "6704877568" 00:57:05 DEBUG 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:57:05 DEBUG ] 00:57:05 DEBUG }, 00:57:05 DEBUG } 00:57:05 DEBUG "value": [ 00:57:05 DEBUG } 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "7.252014301165669" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0.055" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-cts-2" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "ds-idrepo-0" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-cts-2" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "instance": "10.142.0.107:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "instance": "10.142.0.43:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "ds-idrepo-0" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "ds-idrepo-1" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "ds-cts-1" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "0.000124625" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "instance": "10.142.0.107:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "instance": "10.142.0.43:9100" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "5627.266666666666" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139025" 00:57:05 INFO [http_cmd]: http status code OK 00:57:05 DEBUG --- status code --- 00:57:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:05 DEBUG --- http response --- 00:57:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:05 DEBUG ----- output ----- 00:57:05 DEBUG { 00:57:05 DEBUG "status": "success", 00:57:05 DEBUG "data": { 00:57:05 DEBUG "resultType": "vector", 00:57:05 DEBUG "result": [ 00:57:05 DEBUG { 00:57:05 DEBUG "metric": { 00:57:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG [...] 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG 00:57:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:57:05 DEBUG }, 00:57:05 DEBUG "value": [ 00:57:05 DEBUG 1713139025, 00:57:05 DEBUG "2470.3" 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG ] 00:57:05 DEBUG } 00:57:05 DEBUG } 00:57:22 INFO 00:57:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:57:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:57:22 INFO [loop_until]: OK (rc = 0) 00:57:22 DEBUG --- stdout --- 00:57:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13906m 8672Mi am-7d597cf7d6-8vr8f 12490m 8679Mi am-7d597cf7d6-9wz9h 17516m 8988Mi ds-cts-0 12286m 8241Mi ds-cts-1 12550m 8516Mi ds-cts-2 13114m 8196Mi ds-idrepo-0 14106m 4581Mi ds-idrepo-1 2211m 4303Mi ds-idrepo-2 2063m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7357m 7185Mi 00:57:22 DEBUG --- stderr --- 00:57:22 DEBUG 00:57:28 INFO 00:57:28 INFO [loop_until]: kubectl --namespace=xlou top node 00:57:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:57:28 INFO [loop_until]: OK (rc = 0) 00:57:28 DEBUG --- stdout --- 00:57:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4867Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11983m 40% 11097Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12232m 40% 10710Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17843m 59% 3592Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 2014m 6% 6831Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11580m 38% 12435Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11770m 39% 10182Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1888m 6% 5980Mi 5% gke-xlou-cdm-ds-0f601751-j488 11406m 38% 10734Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2229Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16979m 56% 10537Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12073m 40% 6501Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7949m 50% 11133Mi 18% 00:57:28 DEBUG --- stderr --- 00:57:28 DEBUG 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 ERROR [http_cmd]: ERROR 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 ERROR --- status code --- 00:58:05 DEBUG --- status code --- 00:58:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 ERROR --- http response --- 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:58:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:58:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "instance": "10.142.0.107:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "instance": "10.142.0.43:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "14259.333333333332" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "1448.7" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG { 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "instance": "10.142.0.107:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "instance": "10.142.0.43:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "13389900800" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "ds-idrepo-0" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "418403" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "instance": "10.142.0.107:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "instance": "10.142.0.43:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0.00040000000000001514" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0.055" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 DEBUG { 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:58:05 DEBUG "data": { 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG }, 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "value": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-cts-2" 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG [...] 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG "pod": "ds-idrepo-0" 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG }, 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG 00:58:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG "0" 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG ] 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG } 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG ] 00:58:05 DEBUG { 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG } 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "1447.0666666666666" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG "data": { 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG "result": [ 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG { 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG "metric": { 00:58:05 DEBUG } 00:58:05 DEBUG "pod": "ds-cts-2" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139085" 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139085" 00:58:05 DEBUG --- http response --- 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG }, 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "ds-cts-1" 00:58:05 DEBUG "pod": "ds-idrepo-0" 00:58:05 DEBUG }, 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "2481.0333333333333" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "7960633344" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "8.031826691787193" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-cts-2" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "ds-idrepo-0" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-cts-2" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "instance": "10.142.0.107:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "instance": "10.142.0.43:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0.0003333333333333558" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "ds-idrepo-0" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "ds-idrepo-1" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "ds-cts-1" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "0.00023333333333333333" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "instance": "10.142.0.107:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "instance": "10.142.0.43:9100" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "5869.566666666667" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139085" 00:58:05 INFO [http_cmd]: http status code OK 00:58:05 DEBUG --- status code --- 00:58:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:05 DEBUG --- http response --- 00:58:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:05 DEBUG ----- output ----- 00:58:05 DEBUG { 00:58:05 DEBUG "status": "success", 00:58:05 DEBUG "data": { 00:58:05 DEBUG "resultType": "vector", 00:58:05 DEBUG "result": [ 00:58:05 DEBUG { 00:58:05 DEBUG "metric": { 00:58:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG [...] 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG 00:58:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:58:05 DEBUG }, 00:58:05 DEBUG "value": [ 00:58:05 DEBUG 1713139085, 00:58:05 DEBUG "1422.2333333333333" 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG ] 00:58:05 DEBUG } 00:58:05 DEBUG } 00:58:22 INFO 00:58:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:58:22 INFO [loop_until]: OK (rc = 0) 00:58:22 DEBUG --- stdout --- 00:58:22 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9141m 8623Mi am-7d597cf7d6-8vr8f 8782m 8700Mi am-7d597cf7d6-9wz9h 12527m 8906Mi ds-cts-0 9404m 8232Mi ds-cts-1 10158m 8490Mi ds-cts-2 9868m 8202Mi ds-idrepo-0 8385m 4582Mi ds-idrepo-1 1336m 4303Mi ds-idrepo-2 1279m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7892m 7532Mi 00:58:22 DEBUG --- stderr --- 00:58:22 DEBUG 00:58:28 INFO 00:58:28 INFO [loop_until]: kubectl --namespace=xlou top node 00:58:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:58:28 INFO [loop_until]: OK (rc = 0) 00:58:28 DEBUG --- stdout --- 00:58:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4880Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9782m 32% 11087Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8628m 28% 10708Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11898m 39% 3628Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1371m 4% 6832Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9144m 30% 12421Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8449m 28% 10181Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1313m 4% 5983Mi 5% gke-xlou-cdm-ds-0f601751-j488 9737m 32% 10738Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2252Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 11307m 37% 10524Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8604m 28% 6499Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8181m 51% 11059Mi 18% 00:58:28 DEBUG --- stderr --- 00:58:28 DEBUG 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG { 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:59:05 DEBUG "data": { 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "data": { 00:59:05 DEBUG "instance": "10.142.0.107:9100" 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG }, 00:59:05 DEBUG "result": [ 00:59:05 DEBUG "value": [ 00:59:05 DEBUG { 00:59:05 DEBUG 00:59:05 DEBUG "metric": { 00:59:05 DEBUG 00:59:05 DEBUG "instance": "10.142.0.107:9100" 00:59:05 DEBUG 00:59:05 DEBUG }, 00:59:05 DEBUG [...] 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "instance": "10.142.0.43:9100" 00:59:05 DEBUG [...] 00:59:05 DEBUG }, 00:59:05 DEBUG 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG 00:59:05 DEBUG "13390636032" 00:59:05 DEBUG "instance": "10.142.0.43:9100" 00:59:05 DEBUG ] 00:59:05 DEBUG }, 00:59:05 DEBUG } 00:59:05 DEBUG "value": [ 00:59:05 DEBUG ] 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG } 00:59:05 DEBUG "13676.333333333332" 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 ERROR [http_cmd]: ERROR 00:59:05 ERROR --- status code --- 00:59:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:59:05 ERROR --- http response --- 00:59:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:59:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-idrepo-0" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "351542.5" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "1372.4" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "8.01228938251637" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "instance": "10.142.0.107:9100" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "instance": "10.142.0.43:9100" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0.00040000000000001514" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0.055" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0.003404902637342144" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "1373.3" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-idrepo-0" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-cts-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139145" 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-idrepo-0" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "2482.9333333333334" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "8145854464" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "4301.359176692969" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-idrepo-0" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "491.583905907768" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "instance": "10.142.0.107:9100" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "instance": "10.142.0.43:9100" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0.0003333333333333262" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-idrepo-0" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "ds-idrepo-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "ds-cts-1" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "0.000611513888888889" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "instance": "10.142.0.107:9100" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "instance": "10.142.0.43:9100" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "5419.9333333333325" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139145" 00:59:05 INFO [http_cmd]: http status code OK 00:59:05 DEBUG --- status code --- 00:59:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:05 DEBUG --- http response --- 00:59:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:05 DEBUG ----- output ----- 00:59:05 DEBUG { 00:59:05 DEBUG "status": "success", 00:59:05 DEBUG "data": { 00:59:05 DEBUG "resultType": "vector", 00:59:05 DEBUG "result": [ 00:59:05 DEBUG { 00:59:05 DEBUG "metric": { 00:59:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG [...] 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG 00:59:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 00:59:05 DEBUG }, 00:59:05 DEBUG "value": [ 00:59:05 DEBUG 1713139145, 00:59:05 DEBUG "1367.5" 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG ] 00:59:05 DEBUG } 00:59:05 DEBUG } 00:59:22 INFO 00:59:22 INFO [loop_until]: kubectl --namespace=xlou top pods 00:59:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:59:23 INFO [loop_until]: OK (rc = 0) 00:59:23 DEBUG --- stdout --- 00:59:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 8492m 8622Mi am-7d597cf7d6-8vr8f 7866m 8702Mi am-7d597cf7d6-9wz9h 10532m 8901Mi ds-cts-0 9528m 8232Mi ds-cts-1 9708m 8490Mi ds-cts-2 9672m 8197Mi ds-idrepo-0 8753m 4582Mi ds-idrepo-1 1297m 4302Mi ds-idrepo-2 1252m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 6m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8047m 7681Mi 00:59:23 DEBUG --- stderr --- 00:59:23 DEBUG 00:59:28 INFO 00:59:28 INFO [loop_until]: kubectl --namespace=xlou top node 00:59:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:59:28 INFO [loop_until]: OK (rc = 0) 00:59:28 DEBUG --- stdout --- 00:59:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4883Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9538m 31% 11096Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8406m 28% 10702Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11010m 36% 3635Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1348m 4% 6831Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9362m 31% 12434Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8289m 27% 10192Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1265m 4% 5969Mi 5% gke-xlou-cdm-ds-0f601751-j488 9171m 30% 10706Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2252Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10971m 36% 10512Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8271m 27% 6508Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8197m 51% 11204Mi 19% 00:59:28 DEBUG --- stderr --- 00:59:28 DEBUG 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-idrepo-0" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "instance": "10.142.0.107:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "instance": "10.142.0.43:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "17167.93333333333" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "instance": "10.142.0.107:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "instance": "10.142.0.43:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "13391381504" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 ERROR [http_cmd]: ERROR 01:00:05 ERROR --- status code --- 01:00:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:00:05 ERROR --- http response --- 01:00:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:00:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "288157" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "1344.6" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139205" 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "8.02500544241474" 01:00:05 DEBUG "0.055" 01:00:05 DEBUG ] 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "instance": "10.142.0.107:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "instance": "10.142.0.43:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0.00030000000000001136" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0.000018987531022946067" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG { 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-idrepo-0" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "1343" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG { 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-cts-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-idrepo-0" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "2353.6666666666665" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "8254603264" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-idrepo-0" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "instance": "10.142.0.107:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "instance": "10.142.0.43:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-idrepo-0" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "ds-idrepo-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "ds-cts-1" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "0.0014418472222222222" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "instance": "10.142.0.107:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "instance": "10.142.0.43:9100" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "5864.066666666667" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139205" 01:00:05 INFO [http_cmd]: http status code OK 01:00:05 DEBUG --- status code --- 01:00:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:05 DEBUG --- http response --- 01:00:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:05 DEBUG ----- output ----- 01:00:05 DEBUG { 01:00:05 DEBUG "status": "success", 01:00:05 DEBUG "data": { 01:00:05 DEBUG "resultType": "vector", 01:00:05 DEBUG "result": [ 01:00:05 DEBUG { 01:00:05 DEBUG "metric": { 01:00:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG [...] 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG 01:00:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:00:05 DEBUG }, 01:00:05 DEBUG "value": [ 01:00:05 DEBUG 1713139205, 01:00:05 DEBUG "1331" 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG ] 01:00:05 DEBUG } 01:00:05 DEBUG } 01:00:23 INFO 01:00:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:00:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:00:23 INFO [loop_until]: OK (rc = 0) 01:00:23 DEBUG --- stdout --- 01:00:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 8217m 8626Mi am-7d597cf7d6-8vr8f 7727m 8704Mi am-7d597cf7d6-9wz9h 9973m 8897Mi ds-cts-0 9073m 8254Mi ds-cts-1 9138m 8490Mi ds-cts-2 9705m 8203Mi ds-idrepo-0 8076m 4582Mi ds-idrepo-1 1252m 4303Mi ds-idrepo-2 1211m 4096Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7900m 7855Mi 01:00:23 DEBUG --- stderr --- 01:00:23 DEBUG 01:00:28 INFO 01:00:28 INFO [loop_until]: kubectl --namespace=xlou top node 01:00:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:00:28 INFO [loop_until]: OK (rc = 0) 01:00:28 DEBUG --- stdout --- 01:00:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4894Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8315m 27% 11118Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7952m 26% 10706Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11242m 37% 3641Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1286m 4% 6827Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 8031m 26% 12428Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7622m 25% 10186Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1201m 4% 5973Mi 5% gke-xlou-cdm-ds-0f601751-j488 8761m 29% 10790Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2247Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10502m 35% 10523Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7814m 26% 6514Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8030m 50% 10835Mi 18% 01:00:28 DEBUG --- stderr --- 01:00:28 DEBUG 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG "instance": "10.142.0.107:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "ds-idrepo-0" 01:01:05 DEBUG "instance": "10.142.0.43:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG "13392247808" 01:01:05 DEBUG ] 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "240100" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139265" 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "instance": "10.142.0.107:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "instance": "10.142.0.43:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "16894.233333333334" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 ERROR [http_cmd]: ERROR 01:01:05 ERROR --- status code --- 01:01:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:01:05 ERROR --- http response --- 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG --- status code --- 01:01:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG }, 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG }, 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG "1273.6333333333332" 01:01:05 DEBUG ] 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0.055" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "8.032795105210703" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "ds-idrepo-0" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "instance": "10.142.0.107:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "instance": "10.142.0.43:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0.00043333333333350765" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-cts-2" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0.00001232014881410019" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "1273.7666666666667" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "ds-cts-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-cts-2" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG }, 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG "value": [ 01:01:05 DEBUG --- http response --- 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:05 DEBUG "0" 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG ] 01:01:05 DEBUG { 01:01:05 DEBUG { 01:01:05 DEBUG } 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG ] 01:01:05 DEBUG "data": { 01:01:05 DEBUG "data": { 01:01:05 DEBUG } 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG } 01:01:05 DEBUG "result": [ 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG }, 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "value": [ 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG { 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG [...] 01:01:05 DEBUG [...] 01:01:05 DEBUG "data": { 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "result": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG { 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG "pod": "ds-idrepo-0" 01:01:05 DEBUG "metric": { 01:01:05 DEBUG }, 01:01:05 DEBUG }, 01:01:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "value": [ 01:01:05 DEBUG }, 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG "1457.1" 01:01:05 DEBUG "0" 01:01:05 DEBUG 01:01:05 DEBUG ] 01:01:05 DEBUG ] 01:01:05 DEBUG 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG 01:01:05 DEBUG ] 01:01:05 DEBUG ] 01:01:05 DEBUG [...] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "8352776192" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-cts-2" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-cts-2" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "ds-idrepo-0" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "instance": "10.142.0.107:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "instance": "10.142.0.43:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0.00033333333333333365" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "ds-idrepo-0" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "ds-idrepo-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "ds-cts-1" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "0.00013333333333333334" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "instance": "10.142.0.107:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "instance": "10.142.0.43:9100" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "5390.333333333334" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139265" 01:01:05 INFO [http_cmd]: http status code OK 01:01:05 DEBUG --- status code --- 01:01:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:05 DEBUG --- http response --- 01:01:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:05 DEBUG ----- output ----- 01:01:05 DEBUG { 01:01:05 DEBUG "status": "success", 01:01:05 DEBUG "data": { 01:01:05 DEBUG "resultType": "vector", 01:01:05 DEBUG "result": [ 01:01:05 DEBUG { 01:01:05 DEBUG "metric": { 01:01:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG [...] 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG 01:01:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:01:05 DEBUG }, 01:01:05 DEBUG "value": [ 01:01:05 DEBUG 1713139265, 01:01:05 DEBUG "1273.8666666666666" 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG ] 01:01:05 DEBUG } 01:01:05 DEBUG } 01:01:23 INFO 01:01:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:01:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:01:23 INFO [loop_until]: OK (rc = 0) 01:01:23 DEBUG --- stdout --- 01:01:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6848m 8623Mi am-7d597cf7d6-8vr8f 6544m 8701Mi am-7d597cf7d6-9wz9h 8797m 8906Mi ds-cts-0 5860m 8256Mi ds-cts-1 6449m 8589Mi ds-cts-2 6347m 8208Mi ds-idrepo-0 7293m 4583Mi ds-idrepo-1 1143m 4303Mi ds-idrepo-2 1086m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7948m 7912Mi 01:01:23 DEBUG --- stderr --- 01:01:23 DEBUG 01:01:28 INFO 01:01:28 INFO [loop_until]: kubectl --namespace=xlou top node 01:01:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:01:28 INFO [loop_until]: OK (rc = 0) 01:01:28 DEBUG --- stdout --- 01:01:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4885Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5862m 19% 11092Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6083m 20% 10698Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 8628m 28% 3117Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1167m 3% 6824Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5686m 19% 12413Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6606m 22% 10189Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1077m 3% 5972Mi 5% gke-xlou-cdm-ds-0f601751-j488 6155m 20% 10705Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2253Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 7990m 26% 10515Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 5824m 19% 6513Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8236m 51% 11294Mi 19% 01:01:28 DEBUG --- stderr --- 01:01:28 DEBUG 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 ERROR [http_cmd]: ERROR 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG --- http response --- 01:02:05 ERROR --- status code --- 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:02:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 ERROR --- http response --- 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG "result": [ 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG "result": [ 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG { 01:02:05 DEBUG { 01:02:05 DEBUG { 01:02:05 DEBUG { 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG "metric": { 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG "data": { 01:02:05 DEBUG "data": { 01:02:05 DEBUG "instance": "10.142.0.107:9100" 01:02:05 DEBUG "data": { 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG }, 01:02:05 DEBUG { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG }, 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG "value": [ 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG "result": [ 01:02:05 DEBUG "value": [ 01:02:05 DEBUG "result": [ 01:02:05 DEBUG 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG "data": { 01:02:05 DEBUG { 01:02:05 DEBUG { 01:02:05 DEBUG 01:02:05 DEBUG { 01:02:05 DEBUG 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG 01:02:05 DEBUG "metric": { 01:02:05 DEBUG 01:02:05 DEBUG { 01:02:05 DEBUG "result": [ 01:02:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:05 DEBUG 01:02:05 DEBUG "instance": "10.142.0.107:9100" 01:02:05 DEBUG [...] 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG { 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG }, 01:02:05 DEBUG }, 01:02:05 DEBUG [...] 01:02:05 DEBUG }, 01:02:05 DEBUG 01:02:05 DEBUG "data": { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG "value": [ 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "result": [ 01:02:05 DEBUG }, 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-idrepo-0" 01:02:05 DEBUG { 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG 01:02:05 DEBUG "instance": "10.142.0.43:9100" 01:02:05 DEBUG 01:02:05 DEBUG }, 01:02:05 DEBUG "metric": { 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG [...] 01:02:05 DEBUG }, 01:02:05 DEBUG [...] 01:02:05 DEBUG "value": [ 01:02:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG { 01:02:05 DEBUG 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG }, 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG 01:02:05 DEBUG "0" 01:02:05 DEBUG "value": [ 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG "data": { 01:02:05 DEBUG 01:02:05 DEBUG "13392966656" 01:02:05 DEBUG 01:02:05 DEBUG ] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:05 DEBUG ] 01:02:05 DEBUG "instance": "10.142.0.43:9100" 01:02:05 DEBUG } 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG }, 01:02:05 DEBUG "result": [ 01:02:05 DEBUG }, 01:02:05 DEBUG } 01:02:05 DEBUG }, 01:02:05 DEBUG ] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "value": [ 01:02:05 DEBUG { 01:02:05 DEBUG "value": [ 01:02:05 DEBUG ] 01:02:05 DEBUG "value": [ 01:02:05 DEBUG } 01:02:05 DEBUG [...] 01:02:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "metric": { 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG } 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG } 01:02:05 DEBUG 01:02:05 DEBUG }, 01:02:05 DEBUG "0.055" 01:02:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:05 DEBUG "40.13333333333333" 01:02:05 DEBUG } 01:02:05 DEBUG "13792.199999999999" 01:02:05 DEBUG 01:02:05 DEBUG "value": [ 01:02:05 DEBUG ] 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG ] 01:02:05 DEBUG 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG 01:02:05 DEBUG } 01:02:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:02:05 DEBUG "212900.5" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG 01:02:05 DEBUG ] 01:02:05 DEBUG }, 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG 01:02:05 DEBUG } 01:02:05 DEBUG "value": [ 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 DEBUG [...] 01:02:05 DEBUG } 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG 01:02:05 DEBUG "8.03134185947932" 01:02:05 DEBUG } 01:02:05 DEBUG 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG 01:02:05 DEBUG } 01:02:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:05 DEBUG ] 01:02:05 DEBUG }, 01:02:05 DEBUG } 01:02:05 DEBUG "value": [ 01:02:05 DEBUG } 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-idrepo-0" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-cts-2" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-8vr8f" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-idrepo-0" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "instance": "10.142.0.107:9100" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "instance": "10.142.0.43:9100" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0.0003333333333330302" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-cts-2" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-8vr8f" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "3.2427811608622763" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "40.13333333333333" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139325" 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-cts-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "1371.5333333333333" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:02:05 DEBUG }, 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG "value": [ 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG "8556822528" 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-cts-2" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-8vr8f" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "302613023.4162521" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-cts-2" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "am-7d597cf7d6-8vr8f" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "1471978.7728026535" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-idrepo-0" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "instance": "10.142.0.107:9100" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "instance": "10.142.0.43:9100" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-idrepo-0" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139325" 01:02:05 INFO [http_cmd]: http status code OK 01:02:05 DEBUG --- status code --- 01:02:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:05 DEBUG --- http response --- 01:02:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:02:05 DEBUG ----- output ----- 01:02:05 DEBUG { 01:02:05 DEBUG "status": "success", 01:02:05 DEBUG "data": { 01:02:05 DEBUG "resultType": "vector", 01:02:05 DEBUG "result": [ 01:02:05 DEBUG { 01:02:05 DEBUG "metric": { 01:02:05 DEBUG "pod": "ds-idrepo-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG [...] 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG 01:02:05 DEBUG "pod": "ds-cts-1" 01:02:05 DEBUG }, 01:02:05 DEBUG "value": [ 01:02:05 DEBUG 1713139325, 01:02:05 DEBUG "0.00006666666666666667" 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG ] 01:02:05 DEBUG } 01:02:05 DEBUG } 01:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139326" 01:02:06 INFO [http_cmd]: http status code OK 01:02:06 DEBUG --- status code --- 01:02:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:06 DEBUG --- http response --- 01:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:06 DEBUG ----- output ----- 01:02:06 DEBUG { 01:02:06 DEBUG "status": "success", 01:02:06 DEBUG "data": { 01:02:06 DEBUG "resultType": "vector", 01:02:06 DEBUG "result": [ 01:02:06 DEBUG { 01:02:06 DEBUG "metric": { 01:02:06 DEBUG "instance": "10.142.0.107:9100" 01:02:06 DEBUG }, 01:02:06 DEBUG "value": [ 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG [...] 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG "instance": "10.142.0.43:9100" 01:02:06 DEBUG }, 01:02:06 DEBUG "value": [ 01:02:06 DEBUG 1713139326, 01:02:06 DEBUG "5499.499999999999" 01:02:06 DEBUG ] 01:02:06 DEBUG } 01:02:06 DEBUG ] 01:02:06 DEBUG } 01:02:06 DEBUG } 01:02:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139326" 01:02:06 INFO [http_cmd]: http status code OK 01:02:06 DEBUG --- status code --- 01:02:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:06 DEBUG --- http response --- 01:02:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:06 DEBUG ----- output ----- 01:02:06 DEBUG { 01:02:06 DEBUG "status": "success", 01:02:06 DEBUG "data": { 01:02:06 DEBUG "resultType": "vector", 01:02:06 DEBUG "result": [ 01:02:06 DEBUG { 01:02:06 DEBUG "metric": { 01:02:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:02:06 DEBUG }, 01:02:06 DEBUG "value": [ 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG [...] 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG 01:02:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:02:06 DEBUG }, 01:02:06 DEBUG "value": [ 01:02:06 DEBUG 1713139326, 01:02:06 DEBUG "40.13333333333333" 01:02:06 DEBUG ] 01:02:06 DEBUG } 01:02:06 DEBUG ] 01:02:06 DEBUG } 01:02:06 DEBUG } 01:02:23 INFO 01:02:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:02:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:02:23 INFO [loop_until]: OK (rc = 0) 01:02:23 DEBUG --- stdout --- 01:02:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1589m 8667Mi am-7d597cf7d6-8vr8f 2374m 8700Mi am-7d597cf7d6-9wz9h 1582m 8700Mi ds-cts-0 3497m 8251Mi ds-cts-1 3500m 8460Mi ds-cts-2 3429m 8201Mi ds-idrepo-0 745m 4582Mi ds-idrepo-1 149m 4302Mi ds-idrepo-2 105m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1494m 16Mi 01:02:23 DEBUG --- stderr --- 01:02:23 DEBUG 01:02:28 INFO 01:02:28 INFO [loop_until]: kubectl --namespace=xlou top node 01:02:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:02:28 INFO [loop_until]: OK (rc = 0) 01:02:28 DEBUG --- stdout --- 01:02:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4893Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3189m 10% 11091Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1240m 4% 10733Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3046Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 150m 0% 6833Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3032m 10% 12423Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1300m 4% 10189Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 166m 0% 5970Mi 5% gke-xlou-cdm-ds-0f601751-j488 3179m 10% 10704Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2254Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1690m 5% 10306Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 813m 2% 6524Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 74m 0% 3895Mi 6% 01:02:28 DEBUG --- stderr --- 01:02:28 DEBUG 01:02:54 WARNING File /tmp/lodemon_data-20240414_182729/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 01:02:54] "GET /monitoring/average?start_time=20240414_235418&stop_time=20240415_000054 HTTP/1.1" 200 - 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 ERROR [http_cmd]: ERROR 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 ERROR --- status code --- 01:03:05 DEBUG --- status code --- 01:03:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 ERROR --- http response --- 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:03:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0.055" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "ds-idrepo-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "instance": "10.142.0.107:9100" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "instance": "10.142.0.43:9100" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "13393593344" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG ----- output ----- 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG { 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG "data": { 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG "result": [ 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG { 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "data": { 01:03:05 DEBUG "data": { 01:03:05 DEBUG }, 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "value": [ 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG "result": [ 01:03:05 DEBUG "result": [ 01:03:05 DEBUG 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG { 01:03:05 DEBUG { 01:03:05 DEBUG 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "instance": "10.142.0.107:9100" 01:03:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:05 DEBUG [...] 01:03:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:05 DEBUG }, 01:03:05 DEBUG }, 01:03:05 DEBUG }, 01:03:05 DEBUG 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG "value": [ 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG }, 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG [...] 01:03:05 DEBUG [...] 01:03:05 DEBUG "value": [ 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "0.025675342542934337" 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG ] 01:03:05 DEBUG 01:03:05 DEBUG "instance": "10.142.0.43:9100" 01:03:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:05 DEBUG } 01:03:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:05 DEBUG }, 01:03:05 DEBUG }, 01:03:05 DEBUG }, 01:03:05 DEBUG ] 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG "value": [ 01:03:05 DEBUG "value": [ 01:03:05 DEBUG } 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG } 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "13673.766666666665" 01:03:05 DEBUG "133368.5" 01:03:05 DEBUG "0" 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG ] 01:03:05 DEBUG ] 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG ] 01:03:05 DEBUG ] 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "111546368" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "ds-idrepo-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "ds-idrepo-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "instance": "10.142.0.107:9100" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "instance": "10.142.0.43:9100" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0.00033333333333350387" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0.7229913570665696" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "ds-idrepo-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-cts-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "1344.9" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "120421497.00628804" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "4617.581481574266" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "ds-idrepo-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "instance": "10.142.0.107:9100" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "instance": "10.142.0.43:9100" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139385" 01:03:05 INFO [http_cmd]: http status code OK 01:03:05 DEBUG --- status code --- 01:03:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:05 DEBUG --- http response --- 01:03:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:05 DEBUG ----- output ----- 01:03:05 DEBUG { 01:03:05 DEBUG "status": "success", 01:03:05 DEBUG "data": { 01:03:05 DEBUG "resultType": "vector", 01:03:05 DEBUG "result": [ 01:03:05 DEBUG { 01:03:05 DEBUG "metric": { 01:03:05 DEBUG "pod": "ds-idrepo-1" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG [...] 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG 01:03:05 DEBUG "pod": "ds-idrepo-0" 01:03:05 DEBUG }, 01:03:05 DEBUG "value": [ 01:03:05 DEBUG 1713139385, 01:03:05 DEBUG "0" 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG ] 01:03:05 DEBUG } 01:03:05 DEBUG } 01:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139386" 01:03:06 INFO [http_cmd]: http status code OK 01:03:06 DEBUG --- status code --- 01:03:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:06 DEBUG --- http response --- 01:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:03:06 DEBUG ----- output ----- 01:03:06 DEBUG { 01:03:06 DEBUG "status": "success", 01:03:06 DEBUG "data": { 01:03:06 DEBUG "resultType": "vector", 01:03:06 DEBUG "result": [ 01:03:06 DEBUG { 01:03:06 DEBUG "metric": { 01:03:06 DEBUG "pod": "ds-idrepo-1" 01:03:06 DEBUG }, 01:03:06 DEBUG "value": [ 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG [...] 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG "pod": "ds-cts-1" 01:03:06 DEBUG }, 01:03:06 DEBUG "value": [ 01:03:06 DEBUG 1713139386, 01:03:06 DEBUG "0.00006666666666666667" 01:03:06 DEBUG ] 01:03:06 DEBUG } 01:03:06 DEBUG ] 01:03:06 DEBUG } 01:03:06 DEBUG } 01:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139386" 01:03:06 INFO [http_cmd]: http status code OK 01:03:06 DEBUG --- status code --- 01:03:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:06 DEBUG --- http response --- 01:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:06 DEBUG ----- output ----- 01:03:06 DEBUG { 01:03:06 DEBUG "status": "success", 01:03:06 DEBUG "data": { 01:03:06 DEBUG "resultType": "vector", 01:03:06 DEBUG "result": [ 01:03:06 DEBUG { 01:03:06 DEBUG "metric": { 01:03:06 DEBUG "instance": "10.142.0.107:9100" 01:03:06 DEBUG }, 01:03:06 DEBUG "value": [ 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG [...] 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG "instance": "10.142.0.43:9100" 01:03:06 DEBUG }, 01:03:06 DEBUG "value": [ 01:03:06 DEBUG 1713139386, 01:03:06 DEBUG "5318.5" 01:03:06 DEBUG ] 01:03:06 DEBUG } 01:03:06 DEBUG ] 01:03:06 DEBUG } 01:03:06 DEBUG } 01:03:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139386" 01:03:06 INFO [http_cmd]: http status code OK 01:03:06 DEBUG --- status code --- 01:03:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:06 DEBUG --- http response --- 01:03:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:06 DEBUG ----- output ----- 01:03:06 DEBUG { 01:03:06 DEBUG "status": "success", 01:03:06 DEBUG "data": { 01:03:06 DEBUG "resultType": "vector", 01:03:06 DEBUG "result": [ 01:03:06 DEBUG { 01:03:06 DEBUG "metric": { 01:03:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:03:06 DEBUG }, 01:03:06 DEBUG "value": [ 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG [...] 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG 01:03:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:03:06 DEBUG }, 01:03:06 DEBUG "value": [ 01:03:06 DEBUG 1713139386, 01:03:06 DEBUG "0" 01:03:06 DEBUG ] 01:03:06 DEBUG } 01:03:06 DEBUG ] 01:03:06 DEBUG } 01:03:06 DEBUG } 01:03:23 INFO 01:03:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:03:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:03:23 INFO [loop_until]: OK (rc = 0) 01:03:23 DEBUG --- stdout --- 01:03:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1204m 8549Mi am-7d597cf7d6-8vr8f 1161m 8674Mi am-7d597cf7d6-9wz9h 1490m 8631Mi ds-cts-0 2908m 8220Mi ds-cts-1 3258m 8450Mi ds-cts-2 2929m 8194Mi ds-idrepo-0 727m 4582Mi ds-idrepo-1 102m 4303Mi ds-idrepo-2 104m 4095Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1015Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3m 16Mi 01:03:23 DEBUG --- stderr --- 01:03:23 DEBUG 01:03:28 INFO 01:03:28 INFO [loop_until]: kubectl --namespace=xlou top node 01:03:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:03:28 INFO [loop_until]: OK (rc = 0) 01:03:28 DEBUG --- stdout --- 01:03:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4886Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2948m 9% 11097Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1228m 4% 10667Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 104m 0% 3090Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 158m 0% 6861Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 2849m 9% 12435Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 68m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1263m 4% 10163Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 167m 0% 5974Mi 5% gke-xlou-cdm-ds-0f601751-j488 3022m 10% 10711Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 88m 0% 2257Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1687m 5% 10257Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 802m 2% 6528Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 77m 0% 3900Mi 6% 01:03:28 DEBUG --- stderr --- 01:03:28 DEBUG 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 ERROR [http_cmd]: ERROR 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 ERROR --- status code --- 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 ERROR --- http response --- 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:04:05 DEBUG { 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG "data": { 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "data": { 01:04:05 DEBUG { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "result": [ 01:04:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG { 01:04:05 DEBUG }, 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "instance": "10.142.0.107:9100" 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "value": [ 01:04:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:05 DEBUG }, 01:04:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:05 DEBUG 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG }, 01:04:05 DEBUG 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG }, 01:04:05 DEBUG 01:04:05 DEBUG "instance": "10.142.0.43:9100" 01:04:05 DEBUG 01:04:05 DEBUG "value": [ 01:04:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:05 DEBUG }, 01:04:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG }, 01:04:05 DEBUG "0.06666666666666667" 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG ] 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "13394383872" 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG } 01:04:05 DEBUG "0.055" 01:04:05 DEBUG ] 01:04:05 DEBUG "55937" 01:04:05 DEBUG ] 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-idrepo-1" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "ds-idrepo-0" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0.03333333333333333" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0.02871747337665666" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG { 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:05 DEBUG "data": { 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "pod": "ds-idrepo-1" 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG }, 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "value": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "instance": "10.142.0.107:9100" 01:04:05 DEBUG 01:04:05 DEBUG "pod": "ds-idrepo-1" 01:04:05 DEBUG }, 01:04:05 DEBUG 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG [...] 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG "pod": "ds-idrepo-0" 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG }, 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG 01:04:05 DEBUG "instance": "10.142.0.43:9100" 01:04:05 DEBUG "0" 01:04:05 DEBUG "pod": "ds-idrepo-0" 01:04:05 DEBUG }, 01:04:05 DEBUG ] 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG } 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG ] 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "435945.36666666664" 01:04:05 DEBUG } 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0.06666666666666667" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-idrepo-1" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "ds-cts-1" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-cts-2" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "113348608" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-cts-2" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "instance": "10.142.0.107:9100" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "instance": "10.142.0.43:9100" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0.0004666666666665264" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-cts-2" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0.000015917507424051765" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "1273.7" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-cts-2" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "pod": "ds-idrepo-1" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "pod": "ds-idrepo-0" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139445" 01:04:05 INFO [http_cmd]: http status code OK 01:04:05 DEBUG --- status code --- 01:04:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:05 DEBUG --- http response --- 01:04:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:05 DEBUG ----- output ----- 01:04:05 DEBUG { 01:04:05 DEBUG "status": "success", 01:04:05 DEBUG "data": { 01:04:05 DEBUG "resultType": "vector", 01:04:05 DEBUG "result": [ 01:04:05 DEBUG { 01:04:05 DEBUG "metric": { 01:04:05 DEBUG "instance": "10.142.0.107:9100" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG [...] 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG 01:04:05 DEBUG "instance": "10.142.0.43:9100" 01:04:05 DEBUG }, 01:04:05 DEBUG "value": [ 01:04:05 DEBUG 1713139445, 01:04:05 DEBUG "0" 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG ] 01:04:05 DEBUG } 01:04:05 DEBUG } 01:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139446" 01:04:06 INFO [http_cmd]: http status code OK 01:04:06 DEBUG --- status code --- 01:04:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:06 DEBUG --- http response --- 01:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:06 DEBUG ----- output ----- 01:04:06 DEBUG { 01:04:06 DEBUG "status": "success", 01:04:06 DEBUG "data": { 01:04:06 DEBUG "resultType": "vector", 01:04:06 DEBUG "result": [ 01:04:06 DEBUG { 01:04:06 DEBUG "metric": { 01:04:06 DEBUG "pod": "ds-idrepo-1" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG [...] 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG "pod": "ds-idrepo-0" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 1713139446, 01:04:06 DEBUG "0" 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG } 01:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139446" 01:04:06 INFO [http_cmd]: http status code OK 01:04:06 DEBUG --- status code --- 01:04:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:06 DEBUG --- http response --- 01:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:04:06 DEBUG ----- output ----- 01:04:06 DEBUG { 01:04:06 DEBUG "status": "success", 01:04:06 DEBUG "data": { 01:04:06 DEBUG "resultType": "vector", 01:04:06 DEBUG "result": [ 01:04:06 DEBUG { 01:04:06 DEBUG "metric": { 01:04:06 DEBUG "pod": "ds-idrepo-1" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG [...] 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG "pod": "ds-cts-1" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 1713139446, 01:04:06 DEBUG "0.00006431944444444446" 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG } 01:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139446" 01:04:06 INFO [http_cmd]: http status code OK 01:04:06 DEBUG --- status code --- 01:04:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:06 DEBUG --- http response --- 01:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:06 DEBUG ----- output ----- 01:04:06 DEBUG { 01:04:06 DEBUG "status": "success", 01:04:06 DEBUG "data": { 01:04:06 DEBUG "resultType": "vector", 01:04:06 DEBUG "result": [ 01:04:06 DEBUG { 01:04:06 DEBUG "metric": { 01:04:06 DEBUG "instance": "10.142.0.107:9100" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG [...] 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG "instance": "10.142.0.43:9100" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 1713139446, 01:04:06 DEBUG "17759.966666666667" 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG } 01:04:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139446" 01:04:06 INFO [http_cmd]: http status code OK 01:04:06 DEBUG --- status code --- 01:04:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:06 DEBUG --- http response --- 01:04:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:06 DEBUG ----- output ----- 01:04:06 DEBUG { 01:04:06 DEBUG "status": "success", 01:04:06 DEBUG "data": { 01:04:06 DEBUG "resultType": "vector", 01:04:06 DEBUG "result": [ 01:04:06 DEBUG { 01:04:06 DEBUG "metric": { 01:04:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG [...] 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG 01:04:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:04:06 DEBUG }, 01:04:06 DEBUG "value": [ 01:04:06 DEBUG 1713139446, 01:04:06 DEBUG "0.06666666666666667" 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG ] 01:04:06 DEBUG } 01:04:06 DEBUG } 01:04:23 INFO 01:04:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:04:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:04:23 INFO [loop_until]: OK (rc = 0) 01:04:23 DEBUG --- stdout --- 01:04:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1178m 8550Mi am-7d597cf7d6-8vr8f 1166m 8672Mi am-7d597cf7d6-9wz9h 1459m 8634Mi ds-cts-0 2835m 8224Mi ds-cts-1 2976m 8459Mi ds-cts-2 3003m 8202Mi ds-idrepo-0 1114m 4602Mi ds-idrepo-1 535m 4329Mi ds-idrepo-2 446m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 11m 1017Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 831m 61Mi 01:04:23 DEBUG --- stderr --- 01:04:23 DEBUG 01:04:29 INFO 01:04:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:04:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:04:29 INFO [loop_until]: OK (rc = 0) 01:04:29 DEBUG --- stdout --- 01:04:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4883Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2712m 9% 11100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 937m 3% 10668Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3081Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 396m 1% 6828Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 2478m 8% 12439Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1074m 3% 10171Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 405m 1% 5940Mi 5% gke-xlou-cdm-ds-0f601751-j488 2797m 9% 10710Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 2257Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1246m 4% 10278Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 907m 3% 6499Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 649m 4% 4219Mi 7% 01:04:29 DEBUG --- stderr --- 01:04:29 DEBUG 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG { 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG { 01:05:05 DEBUG "data": { 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "resultType": "vector", 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG "data": { 01:05:05 DEBUG "result": [ 01:05:05 DEBUG --- status code --- 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 ERROR [http_cmd]: ERROR 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG { 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 ERROR --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG "result": [ 01:05:05 DEBUG "metric": { 01:05:05 DEBUG --- http response --- 01:05:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG { 01:05:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:05 ERROR --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:05 DEBUG "metric": { 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG }, 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:05:05 DEBUG "instance": "10.142.0.107:9100" 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG "value": [ 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG { 01:05:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:05:05 DEBUG }, 01:05:05 DEBUG { 01:05:05 DEBUG 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG "data": { 01:05:05 DEBUG 01:05:05 DEBUG "data": { 01:05:05 DEBUG 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG [...] 01:05:05 DEBUG "result": [ 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG 01:05:05 DEBUG "result": [ 01:05:05 DEBUG 01:05:05 DEBUG { 01:05:05 DEBUG { 01:05:05 DEBUG [...] 01:05:05 DEBUG { 01:05:05 DEBUG 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG 01:05:05 DEBUG "metric": { 01:05:05 DEBUG 01:05:05 DEBUG "pod": "ds-idrepo-1" 01:05:05 DEBUG "data": { 01:05:05 DEBUG 01:05:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:05 DEBUG }, 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG 01:05:05 DEBUG }, 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "result": [ 01:05:05 DEBUG "instance": "10.142.0.43:9100" 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG { 01:05:05 DEBUG }, 01:05:05 DEBUG 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG "0" 01:05:05 DEBUG 01:05:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG 01:05:05 DEBUG ] 01:05:05 DEBUG [...] 01:05:05 DEBUG }, 01:05:05 DEBUG "13395026944" 01:05:05 DEBUG [...] 01:05:05 DEBUG } 01:05:05 DEBUG 01:05:05 DEBUG "value": [ 01:05:05 DEBUG ] 01:05:05 DEBUG 01:05:05 DEBUG ] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG } 01:05:05 DEBUG 01:05:05 DEBUG } 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG ] 01:05:05 DEBUG 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG "pod": "ds-idrepo-0" 01:05:05 DEBUG 01:05:05 DEBUG } 01:05:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG }, 01:05:05 DEBUG [...] 01:05:05 DEBUG } 01:05:05 DEBUG }, 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG "value": [ 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG "0" 01:05:05 DEBUG 01:05:05 DEBUG "0.055" 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG ] 01:05:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:05 DEBUG ] 01:05:05 DEBUG { 01:05:05 DEBUG } 01:05:05 DEBUG }, 01:05:05 DEBUG } 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG ] 01:05:05 DEBUG "value": [ 01:05:05 DEBUG ] 01:05:05 DEBUG "data": { 01:05:05 DEBUG } 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG } 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG } 01:05:05 DEBUG "639.5" 01:05:05 DEBUG } 01:05:05 DEBUG "result": [ 01:05:05 DEBUG ] 01:05:05 DEBUG { 01:05:05 DEBUG } 01:05:05 DEBUG "metric": { 01:05:05 DEBUG ] 01:05:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:05 DEBUG } 01:05:05 DEBUG }, 01:05:05 DEBUG } 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "ds-idrepo-1" 01:05:05 DEBUG "pod": "ds-idrepo-1" 01:05:05 DEBUG }, 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "ds-idrepo-0" 01:05:05 DEBUG "pod": "ds-idrepo-0" 01:05:05 DEBUG }, 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139505" 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "instance": "10.142.0.107:9100" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "instance": "10.142.0.43:9100" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "13661.7" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "instance": "10.142.0.107:9100" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "instance": "10.142.0.43:9100" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0.0003666666666669964" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0.9423639146473829" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "ds-cts-2" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG { 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:05 DEBUG "metric": { 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:05 DEBUG { 01:05:05 DEBUG }, 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "data": { 01:05:05 DEBUG 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG 01:05:05 DEBUG "result": [ 01:05:05 DEBUG 01:05:05 DEBUG { 01:05:05 DEBUG [...] 01:05:05 DEBUG "metric": { 01:05:05 DEBUG 01:05:05 DEBUG "pod": "ds-idrepo-1" 01:05:05 DEBUG 01:05:05 DEBUG }, 01:05:05 DEBUG 01:05:05 DEBUG "value": [ 01:05:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:05 DEBUG 01:05:05 DEBUG }, 01:05:05 DEBUG 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG [...] 01:05:05 DEBUG "0" 01:05:05 DEBUG 01:05:05 DEBUG ] 01:05:05 DEBUG 01:05:05 DEBUG } 01:05:05 DEBUG 01:05:05 DEBUG ] 01:05:05 DEBUG "pod": "ds-cts-1" 01:05:05 DEBUG } 01:05:05 DEBUG }, 01:05:05 DEBUG } 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "ds-cts-2" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "491597824" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "ds-cts-2" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "42.56666666666666" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "ds-cts-2" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139505" 01:05:05 INFO [http_cmd]: http status code OK 01:05:05 DEBUG --- status code --- 01:05:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:05 DEBUG --- http response --- 01:05:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:05 DEBUG ----- output ----- 01:05:05 DEBUG { 01:05:05 DEBUG "status": "success", 01:05:05 DEBUG "data": { 01:05:05 DEBUG "resultType": "vector", 01:05:05 DEBUG "result": [ 01:05:05 DEBUG { 01:05:05 DEBUG "metric": { 01:05:05 DEBUG "pod": "ds-idrepo-1" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG [...] 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG 01:05:05 DEBUG "pod": "ds-idrepo-0" 01:05:05 DEBUG }, 01:05:05 DEBUG "value": [ 01:05:05 DEBUG 1713139505, 01:05:05 DEBUG "0" 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG ] 01:05:05 DEBUG } 01:05:05 DEBUG } 01:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139506" 01:05:06 INFO [http_cmd]: http status code OK 01:05:06 DEBUG --- status code --- 01:05:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:06 DEBUG --- http response --- 01:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:06 DEBUG ----- output ----- 01:05:06 DEBUG { 01:05:06 DEBUG "status": "success", 01:05:06 DEBUG "data": { 01:05:06 DEBUG "resultType": "vector", 01:05:06 DEBUG "result": [ 01:05:06 DEBUG { 01:05:06 DEBUG "metric": { 01:05:06 DEBUG "instance": "10.142.0.107:9100" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG [...] 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG "instance": "10.142.0.43:9100" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 1713139506, 01:05:06 DEBUG "0" 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG } 01:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139506" 01:05:06 INFO [http_cmd]: http status code OK 01:05:06 DEBUG --- status code --- 01:05:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:06 DEBUG --- http response --- 01:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:06 DEBUG ----- output ----- 01:05:06 DEBUG { 01:05:06 DEBUG "status": "success", 01:05:06 DEBUG "data": { 01:05:06 DEBUG "resultType": "vector", 01:05:06 DEBUG "result": [ 01:05:06 DEBUG { 01:05:06 DEBUG "metric": { 01:05:06 DEBUG "pod": "ds-idrepo-1" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG [...] 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG "pod": "ds-idrepo-0" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 1713139506, 01:05:06 DEBUG "0" 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG } 01:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139506" 01:05:06 INFO [http_cmd]: http status code OK 01:05:06 DEBUG --- status code --- 01:05:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:06 DEBUG --- http response --- 01:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:05:06 DEBUG ----- output ----- 01:05:06 DEBUG { 01:05:06 DEBUG "status": "success", 01:05:06 DEBUG "data": { 01:05:06 DEBUG "resultType": "vector", 01:05:06 DEBUG "result": [ 01:05:06 DEBUG { 01:05:06 DEBUG "metric": { 01:05:06 DEBUG "pod": "ds-idrepo-1" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG [...] 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG "pod": "ds-cts-1" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 1713139506, 01:05:06 DEBUG "0" 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG } 01:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139506" 01:05:06 INFO [http_cmd]: http status code OK 01:05:06 DEBUG --- status code --- 01:05:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:06 DEBUG --- http response --- 01:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:06 DEBUG ----- output ----- 01:05:06 DEBUG { 01:05:06 DEBUG "status": "success", 01:05:06 DEBUG "data": { 01:05:06 DEBUG "resultType": "vector", 01:05:06 DEBUG "result": [ 01:05:06 DEBUG { 01:05:06 DEBUG "metric": { 01:05:06 DEBUG "instance": "10.142.0.107:9100" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG [...] 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG "instance": "10.142.0.43:9100" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 1713139506, 01:05:06 DEBUG "5363.299999999999" 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG } 01:05:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139506" 01:05:06 INFO [http_cmd]: http status code OK 01:05:06 DEBUG --- status code --- 01:05:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:06 DEBUG --- http response --- 01:05:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:06 DEBUG ----- output ----- 01:05:06 DEBUG { 01:05:06 DEBUG "status": "success", 01:05:06 DEBUG "data": { 01:05:06 DEBUG "resultType": "vector", 01:05:06 DEBUG "result": [ 01:05:06 DEBUG { 01:05:06 DEBUG "metric": { 01:05:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG [...] 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG 01:05:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:05:06 DEBUG }, 01:05:06 DEBUG "value": [ 01:05:06 DEBUG 1713139506, 01:05:06 DEBUG "0" 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG ] 01:05:06 DEBUG } 01:05:06 DEBUG } 01:05:23 INFO 01:05:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:05:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:05:23 INFO [loop_until]: OK (rc = 0) 01:05:23 DEBUG --- stdout --- 01:05:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 57m 8548Mi am-7d597cf7d6-8vr8f 122m 8671Mi am-7d597cf7d6-9wz9h 6m 8626Mi ds-cts-0 14m 8197Mi ds-cts-1 18m 8431Mi ds-cts-2 437m 8173Mi ds-idrepo-0 102m 4605Mi ds-idrepo-1 10m 4328Mi ds-idrepo-2 69m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1017Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1050m 457Mi 01:05:23 DEBUG --- stderr --- 01:05:23 DEBUG 01:05:29 INFO 01:05:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:05:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:05:29 INFO [loop_until]: OK (rc = 0) 01:05:29 DEBUG --- stdout --- 01:05:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4887Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 77m 0% 11082Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10671Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3085Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 6860Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 12422Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 64m 0% 10162Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5961Mi 5% gke-xlou-cdm-ds-0f601751-j488 63m 0% 10680Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 2242Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10262Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6502Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 4500Mi 7% 01:05:29 DEBUG --- stderr --- 01:05:29 DEBUG 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "pod": "ds-idrepo-1" 01:06:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:05 DEBUG }, 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "ds-idrepo-0" 01:06:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:05 DEBUG }, 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "0" 01:06:05 DEBUG "0" 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 ERROR [http_cmd]: ERROR 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 ERROR --- status code --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG --- http response --- 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 ERROR --- http response --- 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:06:05 DEBUG --- http response --- 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG { 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG --- http response --- 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG "data": { 01:06:05 DEBUG "data": { 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG "data": { 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG { 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG { 01:06:05 DEBUG "data": { 01:06:05 DEBUG "data": { 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG "data": { 01:06:05 DEBUG "data": { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:05 DEBUG { 01:06:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "data": { 01:06:05 DEBUG "instance": "10.142.0.107:9100" 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "result": [ 01:06:05 DEBUG }, 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG }, 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG }, 01:06:05 DEBUG { 01:06:05 DEBUG { 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "data": { 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "result": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG 01:06:05 DEBUG "instance": "10.142.0.107:9100" 01:06:05 DEBUG "instance": "10.142.0.107:9100" 01:06:05 DEBUG { 01:06:05 DEBUG 01:06:05 DEBUG "pod": "ds-idrepo-1" 01:06:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:05 DEBUG 01:06:05 DEBUG "result": [ 01:06:05 DEBUG 01:06:05 DEBUG }, 01:06:05 DEBUG }, 01:06:05 DEBUG "metric": { 01:06:05 DEBUG 01:06:05 DEBUG }, 01:06:05 DEBUG }, 01:06:05 DEBUG 01:06:05 DEBUG { 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG [...] 01:06:05 DEBUG "metric": { 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG }, 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "ds-idrepo-1" 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG }, 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG [...] 01:06:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 DEBUG 01:06:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "instance": "10.142.0.43:9100" 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG }, 01:06:05 DEBUG 01:06:05 DEBUG }, 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG }, 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG [...] 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "instance": "10.142.0.43:9100" 01:06:05 DEBUG "instance": "10.142.0.43:9100" 01:06:05 DEBUG 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "pod": "ds-idrepo-0" 01:06:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:05 DEBUG "0" 01:06:05 DEBUG 01:06:05 DEBUG "1.0240534360519085" 01:06:05 DEBUG }, 01:06:05 DEBUG }, 01:06:05 DEBUG 01:06:05 DEBUG "13396399104" 01:06:05 DEBUG }, 01:06:05 DEBUG }, 01:06:05 DEBUG ] 01:06:05 DEBUG 01:06:05 DEBUG ] 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:05 DEBUG ] 01:06:05 DEBUG "value": [ 01:06:05 DEBUG "value": [ 01:06:05 DEBUG } 01:06:05 DEBUG 01:06:05 DEBUG } 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG }, 01:06:05 DEBUG } 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG ] 01:06:05 DEBUG "pod": "ds-idrepo-0" 01:06:05 DEBUG ] 01:06:05 DEBUG "0.001599999999999587" 01:06:05 DEBUG "14246.233333333332" 01:06:05 DEBUG "value": [ 01:06:05 DEBUG ] 01:06:05 DEBUG "0" 01:06:05 DEBUG "0.055" 01:06:05 DEBUG } 01:06:05 DEBUG }, 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG "value": [ 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG "1" 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG ] 01:06:05 DEBUG "0" 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "pod": "ds-cts-2" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "0" 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "0" 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "pod": "ds-idrepo-1" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "ds-cts-1" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "0" 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "pod": "ds-cts-2" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "0.0019047303571930092" 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139565" 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139565" 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "768450560" 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:05 INFO [http_cmd]: http status code OK 01:06:05 DEBUG --- status code --- 01:06:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:05 DEBUG --- http response --- 01:06:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:06:05 DEBUG ----- output ----- 01:06:05 DEBUG { 01:06:05 DEBUG "status": "success", 01:06:05 DEBUG "data": { 01:06:05 DEBUG "resultType": "vector", 01:06:05 DEBUG "result": [ 01:06:05 DEBUG { 01:06:05 DEBUG "metric": { 01:06:05 DEBUG "pod": "ds-cts-2" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG [...] 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG 01:06:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:06:05 DEBUG }, 01:06:05 DEBUG "value": [ 01:06:05 DEBUG 1713139565, 01:06:05 DEBUG "0" 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG ] 01:06:05 DEBUG } 01:06:05 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139566" 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "0" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "pod": "ds-cts-2" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "107.95443738528871" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "pod": "ds-idrepo-1" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "pod": "ds-idrepo-0" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "0" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "instance": "10.142.0.107:9100" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "instance": "10.142.0.43:9100" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "0.0009999999999999935" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "pod": "ds-idrepo-1" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "pod": "ds-idrepo-0" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "0" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "pod": "ds-idrepo-1" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "pod": "ds-cts-1" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "0" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "instance": "10.142.0.107:9100" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "instance": "10.142.0.43:9100" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "7894.033333333333" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139566" 01:06:06 INFO [http_cmd]: http status code OK 01:06:06 DEBUG --- status code --- 01:06:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:06 DEBUG --- http response --- 01:06:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:06 DEBUG ----- output ----- 01:06:06 DEBUG { 01:06:06 DEBUG "status": "success", 01:06:06 DEBUG "data": { 01:06:06 DEBUG "resultType": "vector", 01:06:06 DEBUG "result": [ 01:06:06 DEBUG { 01:06:06 DEBUG "metric": { 01:06:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG [...] 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG 01:06:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:06:06 DEBUG }, 01:06:06 DEBUG "value": [ 01:06:06 DEBUG 1713139566, 01:06:06 DEBUG "0" 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG ] 01:06:06 DEBUG } 01:06:06 DEBUG } 01:06:23 INFO 01:06:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:06:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:06:23 INFO [loop_until]: OK (rc = 0) 01:06:23 DEBUG --- stdout --- 01:06:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7m 8538Mi am-7d597cf7d6-8vr8f 7m 8660Mi am-7d597cf7d6-9wz9h 5m 8610Mi ds-cts-0 13m 8196Mi ds-cts-1 12m 8431Mi ds-cts-2 14m 8173Mi ds-idrepo-0 13m 4605Mi ds-idrepo-1 38m 4328Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1017Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1020m 761Mi 01:06:23 DEBUG --- stderr --- 01:06:23 DEBUG 01:06:29 INFO 01:06:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:06:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:06:29 INFO [loop_until]: OK (rc = 0) 01:06:29 DEBUG --- stdout --- 01:06:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4889Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 11062Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10658Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3087Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 6847Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 12411Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10151Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 74m 0% 5960Mi 5% gke-xlou-cdm-ds-0f601751-j488 66m 0% 10685Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 74m 0% 2262Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 85m 0% 10243Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6509Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 477m 3% 4536Mi 7% 01:06:29 DEBUG --- stderr --- 01:06:29 DEBUG 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139625" 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- status code --- 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:05 DEBUG --- http response --- 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG { 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG { 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG { 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "data": { 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:07:05 DEBUG "result": [ 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "result": [ 01:07:05 DEBUG "result": [ 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG { 01:07:05 DEBUG { 01:07:05 DEBUG "data": { 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "pod": "ds-idrepo-1" 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "pod": "ds-idrepo-1" 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "instance": "10.142.0.107:9100" 01:07:05 DEBUG }, 01:07:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:05 DEBUG "result": [ 01:07:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:07:05 DEBUG }, 01:07:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG }, 01:07:05 DEBUG }, 01:07:05 DEBUG { 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG "value": [ 01:07:05 DEBUG "value": [ 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "pod": "ds-idrepo-1" 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG 01:07:05 DEBUG }, 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139625" 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "pod": "ds-idrepo-0" 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG "pod": "ds-idrepo-0" 01:07:05 DEBUG 01:07:05 DEBUG "pod": "ds-cts-1" 01:07:05 DEBUG }, 01:07:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:05 DEBUG { 01:07:05 DEBUG 01:07:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:07:05 DEBUG }, 01:07:05 DEBUG "instance": "10.142.0.43:9100" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG }, 01:07:05 DEBUG }, 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG "value": [ 01:07:05 DEBUG "data": { 01:07:05 DEBUG 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "0" 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "pod": "ds-idrepo-0" 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "0" 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "18874.96349180594" 01:07:05 DEBUG ] 01:07:05 DEBUG "0.055" 01:07:05 DEBUG "0" 01:07:05 DEBUG "result": [ 01:07:05 DEBUG }, 01:07:05 DEBUG "0.5075021595217233" 01:07:05 DEBUG ] 01:07:05 DEBUG "0.0004666666666665264" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG ] 01:07:05 DEBUG { 01:07:05 DEBUG "value": [ 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG "metric": { 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG ] 01:07:05 DEBUG "instance": "10.142.0.107:9100" 01:07:05 DEBUG "0" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG }, 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG "value": [ 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG 01:07:05 DEBUG } 01:07:05 DEBUG 01:07:05 DEBUG } 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "instance": "10.142.0.43:9100" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "13625.766666666665" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 ERROR [http_cmd]: ERROR 01:07:05 ERROR --- status code --- 01:07:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:07:05 ERROR --- http response --- 01:07:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:07:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "instance": "10.142.0.107:9100" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "instance": "10.142.0.43:9100" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "13411163136" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- http response --- 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG { 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "0" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139625" 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "0.5" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "pod": "ds-idrepo-1" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "pod": "ds-cts-1" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "0" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:05 DEBUG "resultType": "vector", 01:07:05 DEBUG "result": [ 01:07:05 DEBUG { 01:07:05 DEBUG "metric": { 01:07:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG [...] 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG 01:07:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:05 DEBUG }, 01:07:05 DEBUG "value": [ 01:07:05 DEBUG 1713139625, 01:07:05 DEBUG "0" 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG ] 01:07:05 DEBUG } 01:07:05 DEBUG } 01:07:05 INFO [http_cmd]: http status code OK 01:07:05 DEBUG --- status code --- 01:07:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:05 DEBUG --- http response --- 01:07:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:05 DEBUG ----- output ----- 01:07:05 DEBUG { 01:07:05 DEBUG "status": "success", 01:07:05 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "ds-cts-1" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139625, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "882221056" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0.03333333333333333" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "ds-idrepo-1" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "ds-idrepo-0" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139626" 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "instance": "10.142.0.107:9100" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "instance": "10.142.0.43:9100" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0.0003333333333333262" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "ds-idrepo-1" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "ds-idrepo-0" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "ds-idrepo-1" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "ds-cts-1" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "instance": "10.142.0.107:9100" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "instance": "10.142.0.43:9100" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "5271.333333333333" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139626" 01:07:06 INFO [http_cmd]: http status code OK 01:07:06 DEBUG --- status code --- 01:07:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:06 DEBUG --- http response --- 01:07:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:06 DEBUG ----- output ----- 01:07:06 DEBUG { 01:07:06 DEBUG "status": "success", 01:07:06 DEBUG "data": { 01:07:06 DEBUG "resultType": "vector", 01:07:06 DEBUG "result": [ 01:07:06 DEBUG { 01:07:06 DEBUG "metric": { 01:07:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG [...] 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG 01:07:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:07:06 DEBUG }, 01:07:06 DEBUG "value": [ 01:07:06 DEBUG 1713139626, 01:07:06 DEBUG "0" 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG ] 01:07:06 DEBUG } 01:07:06 DEBUG } 01:07:23 INFO 01:07:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:07:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:07:23 INFO [loop_until]: OK (rc = 0) 01:07:23 DEBUG --- stdout --- 01:07:23 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8538Mi am-7d597cf7d6-8vr8f 8m 8661Mi am-7d597cf7d6-9wz9h 6m 8609Mi ds-cts-0 16m 8196Mi ds-cts-1 14m 8429Mi ds-cts-2 15m 8173Mi ds-idrepo-0 10m 4605Mi ds-idrepo-1 10m 4328Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1017Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1075m 934Mi 01:07:23 DEBUG --- stderr --- 01:07:23 DEBUG 01:07:29 INFO 01:07:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:07:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:07:29 INFO [loop_until]: OK (rc = 0) 01:07:29 DEBUG --- stdout --- 01:07:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4882Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 11047Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10635Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3095Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6846Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 12422Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2939Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 10137Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 5961Mi 5% gke-xlou-cdm-ds-0f601751-j488 66m 0% 10672Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 79m 0% 2346Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10225Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 63m 0% 6521Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 4984Mi 8% 01:07:29 DEBUG --- stderr --- 01:07:29 DEBUG 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 ERROR [http_cmd]: ERROR 01:08:06 DEBUG --- status code --- 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 ERROR --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 ERROR --- http response --- 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG { 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "data": { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "data": { 01:08:06 DEBUG "data": { 01:08:06 DEBUG "data": { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "data": { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG "data": { 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:08:06 DEBUG "pod": "ds-cts-2" 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "instance": "10.142.0.107:9100" 01:08:06 DEBUG "data": { 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG "instance": "10.142.0.107:9100" 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG "pod": "ds-cts-2" 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG }, 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "result": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG { 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "metric": { 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "instance": "10.142.0.107:9100" 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG }, 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "ds-idrepo-0" 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG 01:08:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:08:06 DEBUG 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG }, 01:08:06 DEBUG "pod": "ds-idrepo-0" 01:08:06 DEBUG 01:08:06 DEBUG "pod": "ds-cts-1" 01:08:06 DEBUG "pod": "ds-idrepo-0" 01:08:06 DEBUG 01:08:06 DEBUG "instance": "10.142.0.43:9100" 01:08:06 DEBUG 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG "instance": "10.142.0.43:9100" 01:08:06 DEBUG }, 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:08:06 DEBUG }, 01:08:06 DEBUG }, 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG }, 01:08:06 DEBUG [...] 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG "value": [ 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG "0" 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "1.029074529506478" 01:08:06 DEBUG "0.055" 01:08:06 DEBUG "0" 01:08:06 DEBUG { 01:08:06 DEBUG "0" 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG "0" 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "13793.3" 01:08:06 DEBUG 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG "0.0004666666666665264" 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG ] 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG "instance": "10.142.0.43:9100" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG "data": { 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG }, 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG ] 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG "value": [ 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG "result": [ 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG { 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG "13411818496" 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG "metric": { 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG "pod": "ds-cts-2" 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 DEBUG }, 01:08:06 DEBUG ] 01:08:06 DEBUG "value": [ 01:08:06 DEBUG } 01:08:06 DEBUG 01:08:06 DEBUG } 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0.00002359784512674361" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139686" 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "1307275264" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "ds-idrepo-0" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "ds-cts-2" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "instance": "10.142.0.107:9100" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "instance": "10.142.0.43:9100" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "ds-idrepo-0" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "ds-idrepo-1" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "ds-cts-1" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "instance": "10.142.0.107:9100" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "instance": "10.142.0.43:9100" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "5458.566666666667" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139686" 01:08:06 INFO [http_cmd]: http status code OK 01:08:06 DEBUG --- status code --- 01:08:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:06 DEBUG --- http response --- 01:08:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:06 DEBUG ----- output ----- 01:08:06 DEBUG { 01:08:06 DEBUG "status": "success", 01:08:06 DEBUG "data": { 01:08:06 DEBUG "resultType": "vector", 01:08:06 DEBUG "result": [ 01:08:06 DEBUG { 01:08:06 DEBUG "metric": { 01:08:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG [...] 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG 01:08:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:08:06 DEBUG }, 01:08:06 DEBUG "value": [ 01:08:06 DEBUG 1713139686, 01:08:06 DEBUG "0" 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG ] 01:08:06 DEBUG } 01:08:06 DEBUG } 01:08:23 INFO 01:08:23 INFO [loop_until]: kubectl --namespace=xlou top pods 01:08:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:08:24 INFO [loop_until]: OK (rc = 0) 01:08:24 DEBUG --- stdout --- 01:08:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8539Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 7m 8609Mi ds-cts-0 15m 8196Mi ds-cts-1 15m 8429Mi ds-cts-2 16m 8174Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 9m 4328Mi ds-idrepo-2 10m 4099Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1017Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1006m 1339Mi 01:08:24 DEBUG --- stderr --- 01:08:24 DEBUG 01:08:29 INFO 01:08:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:08:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:08:29 INFO [loop_until]: OK (rc = 0) 01:08:29 DEBUG --- stdout --- 01:08:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4890Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 76m 0% 11046Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 10637Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3089Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6859Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 12427Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2952Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10142Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 66m 0% 5967Mi 5% gke-xlou-cdm-ds-0f601751-j488 67m 0% 10664Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 82m 0% 2401Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 10229Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 67m 0% 6523Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 5384Mi 9% 01:08:29 DEBUG --- stderr --- 01:08:29 DEBUG 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139746" 01:09:06 ERROR [http_cmd]: ERROR 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 ERROR --- status code --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG --- status code --- 01:09:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- status code --- 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139746" 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 ERROR --- http response --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG --- http response --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG --- status code --- 01:09:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG --- status code --- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG ----- output ----- 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 DEBUG { 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG { 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG { 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG { 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG { 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG { 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG { 01:09:06 DEBUG "data": { 01:09:06 DEBUG { 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "data": { 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG "data": { 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG "data": { 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "data": { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "data": { 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "data": { 01:09:06 DEBUG "data": { 01:09:06 DEBUG { 01:09:06 DEBUG "result": [ 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG "result": [ 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG { 01:09:06 DEBUG { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "metric": { 01:09:06 DEBUG { 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:09:06 DEBUG { 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "instance": "10.142.0.107:9100" 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:09:06 DEBUG "data": { 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "metric": { 01:09:06 DEBUG { 01:09:06 DEBUG "data": { 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG { 01:09:06 DEBUG { 01:09:06 DEBUG }, 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG "data": { 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG { 01:09:06 DEBUG "data": { 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "instance": "10.142.0.107:9100" 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG }, 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "value": [ 01:09:06 DEBUG }, 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG { 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG }, 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG { 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "data": { 01:09:06 DEBUG "result": [ 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG { 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG }, 01:09:06 DEBUG { 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG { 01:09:06 DEBUG "data": { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "metric": { 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "result": [ 01:09:06 DEBUG }, 01:09:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:09:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "instance": "10.142.0.107:9100" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG "result": [ 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG { 01:09:06 DEBUG "value": [ 01:09:06 DEBUG }, 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG }, 01:09:06 DEBUG { 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "metric": { 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "metric": { 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG }, 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG "instance": "10.142.0.43:9100" 01:09:06 DEBUG 01:09:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG }, 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG }, 01:09:06 DEBUG }, 01:09:06 DEBUG [...] 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "instance": "10.142.0.43:9100" 01:09:06 DEBUG }, 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG }, 01:09:06 DEBUG 01:09:06 DEBUG "pod": "ds-idrepo-0" 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "pod": "ds-cts-1" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0" 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "pod": "ds-idrepo-0" 01:09:06 DEBUG "value": [ 01:09:06 DEBUG [...] 01:09:06 DEBUG }, 01:09:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:09:06 DEBUG }, 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0" 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "13412547584" 01:09:06 DEBUG 01:09:06 DEBUG "0" 01:09:06 DEBUG "0" 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG 01:09:06 DEBUG ] 01:09:06 DEBUG [...] 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG }, 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG ] 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG ] 01:09:06 DEBUG 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0.00036666666666652263" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG 01:09:06 DEBUG "0.055" 01:09:06 DEBUG "value": [ 01:09:06 DEBUG "0" 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "1.0299932190778966" 01:09:06 DEBUG } 01:09:06 DEBUG "1749184512" 01:09:06 DEBUG } 01:09:06 DEBUG "instance": "10.142.0.43:9100" 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "pod": "ds-idrepo-0" 01:09:06 DEBUG }, 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG }, 01:09:06 DEBUG ] 01:09:06 DEBUG 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG "0" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG "value": [ 01:09:06 DEBUG } 01:09:06 DEBUG "pod": "ds-idrepo-0" 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG "0" 01:09:06 DEBUG }, 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "13955.999999999998" 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG "value": [ 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG }, 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG "value": [ 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG "0.0028754780370984392" 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG "0" 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "38316975.44587012" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "instance": "10.142.0.107:9100" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "instance": "10.142.0.43:9100" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0.0003333333333333262" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "pod": "ds-idrepo-0" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "pod": "ds-idrepo-1" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "pod": "ds-cts-1" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "instance": "10.142.0.107:9100" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "instance": "10.142.0.43:9100" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "6129.633333333333" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139746" 01:09:06 INFO [http_cmd]: http status code OK 01:09:06 DEBUG --- status code --- 01:09:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:06 DEBUG --- http response --- 01:09:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:06 DEBUG ----- output ----- 01:09:06 DEBUG { 01:09:06 DEBUG "status": "success", 01:09:06 DEBUG "data": { 01:09:06 DEBUG "resultType": "vector", 01:09:06 DEBUG "result": [ 01:09:06 DEBUG { 01:09:06 DEBUG "metric": { 01:09:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG [...] 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG 01:09:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:09:06 DEBUG }, 01:09:06 DEBUG "value": [ 01:09:06 DEBUG 1713139746, 01:09:06 DEBUG "0" 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG ] 01:09:06 DEBUG } 01:09:06 DEBUG } 01:09:24 INFO 01:09:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:09:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:09:24 INFO [loop_until]: OK (rc = 0) 01:09:24 DEBUG --- stdout --- 01:09:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 6m 8610Mi ds-cts-0 16m 8197Mi ds-cts-1 20m 8429Mi ds-cts-2 17m 8175Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 12m 4328Mi ds-idrepo-2 11m 4099Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1017Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1001m 1705Mi 01:09:24 DEBUG --- stderr --- 01:09:24 DEBUG 01:09:29 INFO 01:09:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:09:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:09:29 INFO [loop_until]: OK (rc = 0) 01:09:29 DEBUG --- stdout --- 01:09:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4884Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 75m 0% 11044Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10628Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3102Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 6863Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 12454Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2939Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 59m 0% 10155Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5961Mi 5% gke-xlou-cdm-ds-0f601751-j488 68m 0% 10668Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 2415Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10219Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 72m 0% 6534Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 5785Mi 9% 01:09:29 DEBUG --- stderr --- 01:09:29 DEBUG 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139806" 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 ERROR [http_cmd]: ERROR 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 DEBUG --- status code --- 01:10:06 ERROR --- status code --- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG --- status code --- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 ERROR --- http response --- 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG ----- output ----- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG { 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG "data": { 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG --- http response --- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG { 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG "result": [ 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG "data": { 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "data": { 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "metric": { 01:10:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG "data": { 01:10:06 DEBUG "data": { 01:10:06 DEBUG { 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG }, 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "metric": { 01:10:06 DEBUG { 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:10:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG }, 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG }, 01:10:06 DEBUG "data": { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:10:06 DEBUG }, 01:10:06 DEBUG }, 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "instance": "10.142.0.107:9100" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG "instance": "10.142.0.107:9100" 01:10:06 DEBUG 01:10:06 DEBUG "result": [ 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG }, 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG { 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG "metric": { 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG [...] 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG [...] 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "ds-idrepo-0" 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG "pod": "ds-idrepo-0" 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG "pod": "ds-idrepo-0" 01:10:06 DEBUG }, 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "idm-b85f9d978-rl76f" 01:10:06 DEBUG }, 01:10:06 DEBUG }, 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "instance": "10.142.0.43:9100" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "0" 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG [...] 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG ] 01:10:06 DEBUG "pod": "ds-idrepo-0" 01:10:06 DEBUG "instance": "10.142.0.43:9100" 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG "0" 01:10:06 DEBUG }, 01:10:06 DEBUG }, 01:10:06 DEBUG "0" 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "1.036043115986829" 01:10:06 DEBUG "0" 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG ] 01:10:06 DEBUG 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG "13413176320" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG } 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG "0" 01:10:06 DEBUG "0.00030000000000001136" 01:10:06 DEBUG ] 01:10:06 DEBUG }, 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG "value": [ 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 DEBUG ] 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139806" 01:10:06 DEBUG ] 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139806" 01:10:06 DEBUG "0" 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG ] 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG } 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG } 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG "data": { 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG "result": [ 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG { 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG "metric": { 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG }, 01:10:06 DEBUG ----- output ----- 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG { 01:10:06 DEBUG "value": [ 01:10:06 DEBUG { 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG "data": { 01:10:06 DEBUG 01:10:06 DEBUG "data": { 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:10:06 DEBUG "result": [ 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG [...] 01:10:06 DEBUG "result": [ 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG 01:10:06 DEBUG { 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG { 01:10:06 DEBUG 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG 01:10:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:10:06 DEBUG "data": { 01:10:06 DEBUG }, 01:10:06 DEBUG "data": { 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG }, 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG "result": [ 01:10:06 DEBUG 01:10:06 DEBUG "result": [ 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG { 01:10:06 DEBUG 01:10:06 DEBUG { 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 01:10:06 DEBUG "metric": { 01:10:06 DEBUG 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "0.055" 01:10:06 DEBUG 01:10:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:10:06 DEBUG [...] 01:10:06 DEBUG "instance": "10.142.0.107:9100" 01:10:06 DEBUG ] 01:10:06 DEBUG [...] 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG } 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG "value": [ 01:10:06 DEBUG ] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG } 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "ds-cts-1" 01:10:06 DEBUG 01:10:06 DEBUG } 01:10:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG 01:10:06 DEBUG }, 01:10:06 DEBUG [...] 01:10:06 DEBUG "value": [ 01:10:06 DEBUG [...] 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG 01:10:06 DEBUG "0" 01:10:06 DEBUG 01:10:06 DEBUG "2108825600" 01:10:06 DEBUG 01:10:06 DEBUG ] 01:10:06 DEBUG 01:10:06 DEBUG ] 01:10:06 DEBUG "pod": "idm-b85f9d978-rl76f" 01:10:06 DEBUG } 01:10:06 DEBUG "instance": "10.142.0.43:9100" 01:10:06 DEBUG } 01:10:06 DEBUG }, 01:10:06 DEBUG ] 01:10:06 DEBUG }, 01:10:06 DEBUG ] 01:10:06 DEBUG "value": [ 01:10:06 DEBUG } 01:10:06 DEBUG "value": [ 01:10:06 DEBUG } 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG } 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG } 01:10:06 DEBUG "818.6620220997629" 01:10:06 DEBUG "14690.999999999998" 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139806" 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "instance": "10.142.0.107:9100" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "instance": "10.142.0.43:9100" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0.00033333333333333365" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "ds-idrepo-0" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "ds-idrepo-1" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "ds-cts-1" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139806" 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "instance": "10.142.0.107:9100" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "instance": "10.142.0.43:9100" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "5534.4" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139806" 01:10:06 INFO [http_cmd]: http status code OK 01:10:06 DEBUG --- status code --- 01:10:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:06 DEBUG --- http response --- 01:10:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:06 DEBUG ----- output ----- 01:10:06 DEBUG { 01:10:06 DEBUG "status": "success", 01:10:06 DEBUG "data": { 01:10:06 DEBUG "resultType": "vector", 01:10:06 DEBUG "result": [ 01:10:06 DEBUG { 01:10:06 DEBUG "metric": { 01:10:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG [...] 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG 01:10:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:10:06 DEBUG }, 01:10:06 DEBUG "value": [ 01:10:06 DEBUG 1713139806, 01:10:06 DEBUG "0" 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG ] 01:10:06 DEBUG } 01:10:06 DEBUG } 01:10:24 INFO 01:10:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:10:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:10:24 INFO [loop_until]: OK (rc = 0) 01:10:24 DEBUG --- stdout --- 01:10:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8539Mi am-7d597cf7d6-8vr8f 8m 8661Mi am-7d597cf7d6-9wz9h 7m 8610Mi ds-cts-0 17m 8196Mi ds-cts-1 14m 8431Mi ds-cts-2 16m 8173Mi ds-idrepo-0 14m 4605Mi ds-idrepo-1 11m 4327Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1017Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1020m 2085Mi 01:10:24 DEBUG --- stderr --- 01:10:24 DEBUG 01:10:29 INFO 01:10:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:10:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:10:29 INFO [loop_until]: OK (rc = 0) 01:10:29 DEBUG --- stdout --- 01:10:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4891Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 77m 0% 11043Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10627Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3106Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 6859Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 12455Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2944Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 65m 0% 10148Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5974Mi 5% gke-xlou-cdm-ds-0f601751-j488 69m 0% 10668Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2418Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 10218Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 65m 0% 6527Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 533m 3% 5929Mi 10% 01:10:29 DEBUG --- stderr --- 01:10:29 DEBUG 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 ERROR [http_cmd]: ERROR 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 ERROR --- status code --- 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 ERROR --- http response --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- http response --- 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG { 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG { 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG { 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG "data": { 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:06 DEBUG "data": { 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:11:06 DEBUG { 01:11:06 DEBUG "data": { 01:11:06 DEBUG { 01:11:06 DEBUG { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG { 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG { 01:11:06 DEBUG { 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "data": { 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "data": { 01:11:06 DEBUG "data": { 01:11:06 DEBUG { 01:11:06 DEBUG "data": { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "data": { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "data": { 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG { 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG { 01:11:06 DEBUG { 01:11:06 DEBUG }, 01:11:06 DEBUG { 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "result": [ 01:11:06 DEBUG }, 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "metric": { 01:11:06 DEBUG }, 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "metric": { 01:11:06 DEBUG { 01:11:06 DEBUG { 01:11:06 DEBUG "value": [ 01:11:06 DEBUG { 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG 01:11:06 DEBUG "pod": "ds-cts-2" 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG 01:11:06 DEBUG "metric": { 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG 01:11:06 DEBUG "pod": "ds-cts-2" 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG "value": [ 01:11:06 DEBUG }, 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "value": [ 01:11:06 DEBUG [...] 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG [...] 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG "pod": "ds-idrepo-0" 01:11:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "pod": "ds-idrepo-0" 01:11:06 DEBUG "pod": "ds-idrepo-0" 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG }, 01:11:06 DEBUG }, 01:11:06 DEBUG "0.6683943602576048" 01:11:06 DEBUG }, 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG "0" 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "0" 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "value": [ 01:11:06 DEBUG ] 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "pod": "ds-idrepo-0" 01:11:06 DEBUG ] 01:11:06 DEBUG }, 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG ] 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG } 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG }, 01:11:06 DEBUG } 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "0" 01:11:06 DEBUG } 01:11:06 DEBUG "0" 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG "0" 01:11:06 DEBUG "0.055" 01:11:06 DEBUG "value": [ 01:11:06 DEBUG ] 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG ] 01:11:06 DEBUG ] 01:11:06 DEBUG ] 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 DEBUG ] 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG } 01:11:06 DEBUG "0" 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG "0" 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "data": { 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG }, 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "value": [ 01:11:06 DEBUG { 01:11:06 DEBUG 01:11:06 DEBUG "metric": { 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG 01:11:06 DEBUG }, 01:11:06 DEBUG [...] 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG [...] 01:11:06 DEBUG }, 01:11:06 DEBUG 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG 01:11:06 DEBUG "0" 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG ] 01:11:06 DEBUG }, 01:11:06 DEBUG } 01:11:06 DEBUG "value": [ 01:11:06 DEBUG ] 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG } 01:11:06 DEBUG "0" 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "instance": "10.142.0.107:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "instance": "10.142.0.43:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "13413766144" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "instance": "10.142.0.107:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "instance": "10.142.0.43:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0.0003333333333330302" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139866" 01:11:06 DEBUG --- status code --- 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139866" 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "ds-cts-1" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "2257190912" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "ds-cts-2" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG }, 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG "value": [ 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG 01:11:06 DEBUG { 01:11:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG }, 01:11:06 DEBUG "data": { 01:11:06 DEBUG "value": [ 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "result": [ 01:11:06 DEBUG "0" 01:11:06 DEBUG { 01:11:06 DEBUG ] 01:11:06 DEBUG "metric": { 01:11:06 DEBUG } 01:11:06 DEBUG "instance": "10.142.0.107:9100" 01:11:06 DEBUG ] 01:11:06 DEBUG }, 01:11:06 DEBUG } 01:11:06 DEBUG "value": [ 01:11:06 DEBUG } 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "instance": "10.142.0.43:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "15134.633333333331" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "ds-cts-2" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139866" 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "instance": "10.142.0.107:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "instance": "10.142.0.43:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "ds-idrepo-0" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "ds-idrepo-1" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "ds-cts-1" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "instance": "10.142.0.107:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "instance": "10.142.0.43:9100" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "9081.666666666666" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139866" 01:11:06 INFO [http_cmd]: http status code OK 01:11:06 DEBUG --- status code --- 01:11:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:06 DEBUG --- http response --- 01:11:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:06 DEBUG ----- output ----- 01:11:06 DEBUG { 01:11:06 DEBUG "status": "success", 01:11:06 DEBUG "data": { 01:11:06 DEBUG "resultType": "vector", 01:11:06 DEBUG "result": [ 01:11:06 DEBUG { 01:11:06 DEBUG "metric": { 01:11:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG [...] 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG 01:11:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:11:06 DEBUG }, 01:11:06 DEBUG "value": [ 01:11:06 DEBUG 1713139866, 01:11:06 DEBUG "0" 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG ] 01:11:06 DEBUG } 01:11:06 DEBUG } 01:11:24 INFO 01:11:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:11:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:11:24 INFO [loop_until]: OK (rc = 0) 01:11:24 DEBUG --- stdout --- 01:11:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 16m 8539Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 7m 8609Mi ds-cts-0 19m 8196Mi ds-cts-1 15m 8431Mi ds-cts-2 17m 8173Mi ds-idrepo-0 11m 4604Mi ds-idrepo-1 11m 4328Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1060m 2281Mi 01:11:24 DEBUG --- stderr --- 01:11:24 DEBUG 01:11:29 INFO 01:11:29 INFO [loop_until]: kubectl --namespace=xlou top node 01:11:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:11:29 INFO [loop_until]: OK (rc = 0) 01:11:29 DEBUG --- stdout --- 01:11:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 75m 0% 11056Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10623Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3097Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6867Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 12470Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 67m 0% 10142Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5980Mi 5% gke-xlou-cdm-ds-0f601751-j488 66m 0% 10662Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2428Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 10224Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6526Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1086m 6% 6416Mi 10% 01:11:29 DEBUG --- stderr --- 01:11:29 DEBUG 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-idrepo-0" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-idrepo-0" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG --- status code --- 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG { 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG "data": { 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG "result": [ 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG { 01:12:06 DEBUG { 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "data": { 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG }, 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG "result": [ 01:12:06 DEBUG "value": [ 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG { 01:12:06 DEBUG 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG 01:12:06 DEBUG "instance": "10.142.0.107:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG }, 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139926" 01:12:06 DEBUG [...] 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG [...] 01:12:06 DEBUG }, 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "0" 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-idrepo-0" 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG ] 01:12:06 DEBUG "instance": "10.142.0.43:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG }, 01:12:06 DEBUG } 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG "value": [ 01:12:06 DEBUG ] 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG } 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG "0" 01:12:06 DEBUG } 01:12:06 DEBUG "13414353920" 01:12:06 DEBUG ] 01:12:06 DEBUG ] 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG ] 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139926" 01:12:06 ERROR [http_cmd]: ERROR 01:12:06 ERROR --- status code --- 01:12:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:12:06 ERROR --- http response --- 01:12:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:12:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG { 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "data": { 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG }, 01:12:06 DEBUG "result": [ 01:12:06 DEBUG "value": [ 01:12:06 DEBUG { 01:12:06 DEBUG 01:12:06 DEBUG "metric": { 01:12:06 DEBUG 01:12:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:12:06 DEBUG 01:12:06 DEBUG }, 01:12:06 DEBUG [...] 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG [...] 01:12:06 DEBUG }, 01:12:06 DEBUG 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG 01:12:06 DEBUG "0" 01:12:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:12:06 DEBUG ] 01:12:06 DEBUG }, 01:12:06 DEBUG } 01:12:06 DEBUG "value": [ 01:12:06 DEBUG ] 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG } 01:12:06 DEBUG "1.0367621036951422" 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "instance": "10.142.0.107:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "instance": "10.142.0.43:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0.0003333333333330302" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG { 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG "data": { 01:12:06 DEBUG { 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG "result": [ 01:12:06 DEBUG "data": { 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG { 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG "metric": { 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG { 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG }, 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "data": { 01:12:06 DEBUG "value": [ 01:12:06 DEBUG "data": { 01:12:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG }, 01:12:06 DEBUG "result": [ 01:12:06 DEBUG 01:12:06 DEBUG "result": [ 01:12:06 DEBUG "value": [ 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG { 01:12:06 DEBUG 01:12:06 DEBUG { 01:12:06 DEBUG 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG "metric": { 01:12:06 DEBUG [...] 01:12:06 DEBUG "metric": { 01:12:06 DEBUG 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG }, 01:12:06 DEBUG 01:12:06 DEBUG }, 01:12:06 DEBUG [...] 01:12:06 DEBUG "value": [ 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:12:06 DEBUG 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG "pod": "ds-idrepo-0" 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG { 01:12:06 DEBUG }, 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-cts-0" 01:12:06 DEBUG [...] 01:12:06 DEBUG "data": { 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG [...] 01:12:06 DEBUG }, 01:12:06 DEBUG 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "0" 01:12:06 DEBUG 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG "result": [ 01:12:06 DEBUG ] 01:12:06 DEBUG 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG 01:12:06 DEBUG { 01:12:06 DEBUG } 01:12:06 DEBUG 01:12:06 DEBUG "0" 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG "metric": { 01:12:06 DEBUG ] 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG ] 01:12:06 DEBUG }, 01:12:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:12:06 DEBUG } 01:12:06 DEBUG }, 01:12:06 DEBUG } 01:12:06 DEBUG "value": [ 01:12:06 DEBUG }, 01:12:06 DEBUG } 01:12:06 DEBUG "value": [ 01:12:06 DEBUG ] 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG } 01:12:06 DEBUG "0.055" 01:12:06 DEBUG 01:12:06 DEBUG "0" 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG [...] 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG 01:12:06 DEBUG } 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-cts-0" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0.015070852078515894" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-cts-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "2811715584" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-cts-0" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "17206.668430801805" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "instance": "10.142.0.107:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "instance": "10.142.0.43:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "14314.066666666666" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-cts-0" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "23032.548293199263" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "instance": "10.142.0.107:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "instance": "10.142.0.43:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0.00033333333333333365" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-idrepo-0" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "ds-idrepo-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "ds-cts-1" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "instance": "10.142.0.107:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "instance": "10.142.0.43:9100" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "7729.166666666666" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139926" 01:12:06 INFO [http_cmd]: http status code OK 01:12:06 DEBUG --- status code --- 01:12:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:06 DEBUG --- http response --- 01:12:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:06 DEBUG ----- output ----- 01:12:06 DEBUG { 01:12:06 DEBUG "status": "success", 01:12:06 DEBUG "data": { 01:12:06 DEBUG "resultType": "vector", 01:12:06 DEBUG "result": [ 01:12:06 DEBUG { 01:12:06 DEBUG "metric": { 01:12:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG [...] 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG 01:12:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:12:06 DEBUG }, 01:12:06 DEBUG "value": [ 01:12:06 DEBUG 1713139926, 01:12:06 DEBUG "0" 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG ] 01:12:06 DEBUG } 01:12:06 DEBUG } 01:12:24 INFO 01:12:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:12:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:12:24 INFO [loop_until]: OK (rc = 0) 01:12:24 DEBUG --- stdout --- 01:12:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8662Mi am-7d597cf7d6-9wz9h 6m 8609Mi ds-cts-0 15m 8198Mi ds-cts-1 11m 8429Mi ds-cts-2 15m 8173Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 11m 4328Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 4m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 996m 2867Mi 01:12:24 DEBUG --- stderr --- 01:12:24 DEBUG 01:12:30 INFO 01:12:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:12:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:12:30 INFO [loop_until]: OK (rc = 0) 01:12:30 DEBUG --- stdout --- 01:12:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4889Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 78m 0% 11061Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10623Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3089Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6871Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 12470Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2948Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 64m 0% 10138Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 68m 0% 5969Mi 5% gke-xlou-cdm-ds-0f601751-j488 66m 0% 10671Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2429Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 10221Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 6521Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 7026Mi 11% 01:12:30 DEBUG --- stderr --- 01:12:30 DEBUG 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:13:06 ERROR [http_cmd]: ERROR 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 ERROR --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG ----- output ----- 01:13:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG { 01:13:06 ERROR --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "status": "success", 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "data": { 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG { 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG "status": "success", 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "metric": { 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:06 DEBUG "metric": { 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 DEBUG "data": { 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG { 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG ----- output ----- 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG "metric": { 01:13:06 DEBUG }, 01:13:06 DEBUG { 01:13:06 DEBUG }, 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG "instance": "10.142.0.107:9100" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG }, 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:13:06 DEBUG 01:13:06 DEBUG "data": { 01:13:06 DEBUG 01:13:06 DEBUG "data": { 01:13:06 DEBUG "data": { 01:13:06 DEBUG "data": { 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG "data": { 01:13:06 DEBUG "data": { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "data": { 01:13:06 DEBUG "value": [ 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG 01:13:06 DEBUG { 01:13:06 DEBUG 01:13:06 DEBUG "result": [ 01:13:06 DEBUG 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "result": [ 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG "result": [ 01:13:06 DEBUG }, 01:13:06 DEBUG "result": [ 01:13:06 DEBUG 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG [...] 01:13:06 DEBUG { 01:13:06 DEBUG [...] 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG "value": [ 01:13:06 DEBUG { 01:13:06 DEBUG 01:13:06 DEBUG "data": { 01:13:06 DEBUG 01:13:06 DEBUG "metric": { 01:13:06 DEBUG 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "metric": { 01:13:06 DEBUG 01:13:06 DEBUG "metric": { 01:13:06 DEBUG [...] 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG "pod": "ds-cts-2" 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG "data": { 01:13:06 DEBUG "instance": "10.142.0.107:9100" 01:13:06 DEBUG 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG 01:13:06 DEBUG "result": [ 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG }, 01:13:06 DEBUG }, 01:13:06 DEBUG }, 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG }, 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG 01:13:06 DEBUG { 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG [...] 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG "metric": { 01:13:06 DEBUG }, 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG { 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "ds-cts-2" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "metric": { 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "instance": "10.142.0.43:9100" 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG [...] 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "0" 01:13:06 DEBUG [...] 01:13:06 DEBUG "0" 01:13:06 DEBUG [...] 01:13:06 DEBUG [...] 01:13:06 DEBUG [...] 01:13:06 DEBUG [...] 01:13:06 DEBUG }, 01:13:06 DEBUG [...] 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG ] 01:13:06 DEBUG 01:13:06 DEBUG ] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713139986" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG }, 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG } 01:13:06 DEBUG 01:13:06 DEBUG } 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "13415078912" 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG ] 01:13:06 DEBUG 01:13:06 DEBUG ] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "instance": "10.142.0.43:9100" 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG [...] 01:13:06 DEBUG } 01:13:06 DEBUG "pod": "ds-idrepo-0" 01:13:06 DEBUG } 01:13:06 DEBUG "pod": "ds-idrepo-0" 01:13:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:13:06 DEBUG }, 01:13:06 DEBUG "pod": "ds-cts-1" 01:13:06 DEBUG 01:13:06 DEBUG ] 01:13:06 DEBUG }, 01:13:06 DEBUG "0" 01:13:06 DEBUG }, 01:13:06 DEBUG 01:13:06 DEBUG } 01:13:06 DEBUG }, 01:13:06 DEBUG } 01:13:06 DEBUG }, 01:13:06 DEBUG "pod": "ds-idrepo-0" 01:13:06 DEBUG "value": [ 01:13:06 DEBUG }, 01:13:06 DEBUG [...] 01:13:06 DEBUG } 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG }, 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG ] 01:13:06 DEBUG ] 01:13:06 DEBUG "0.00043333333333303397" 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "0.001751061400557606" 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG "0.055" 01:13:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:13:06 DEBUG "0" 01:13:06 DEBUG "0" 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG ] 01:13:06 DEBUG "0" 01:13:06 DEBUG 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG ] 01:13:06 DEBUG "0" 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG }, 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG "value": [ 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG "0" 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG ] 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "1.0317464656363446" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "ds-idrepo-0" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713139986" 01:13:06 DEBUG 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "3433844736" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "ds-cts-2" 01:13:06 DEBUG "instance": "10.142.0.107:9100" 01:13:06 DEBUG }, 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:13:06 DEBUG "instance": "10.142.0.43:9100" 01:13:06 DEBUG }, 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "333.19978459881526" 01:13:06 DEBUG "14347.93333333333" 01:13:06 DEBUG ] 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "ds-cts-2" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "instance": "10.142.0.107:9100" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "instance": "10.142.0.43:9100" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0.00033333333333333365" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "ds-idrepo-0" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "ds-idrepo-1" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "ds-cts-1" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "instance": "10.142.0.107:9100" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "instance": "10.142.0.43:9100" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "8831.066666666666" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713139986" 01:13:06 INFO [http_cmd]: http status code OK 01:13:06 DEBUG --- status code --- 01:13:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:06 DEBUG --- http response --- 01:13:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:06 DEBUG ----- output ----- 01:13:06 DEBUG { 01:13:06 DEBUG "status": "success", 01:13:06 DEBUG "data": { 01:13:06 DEBUG "resultType": "vector", 01:13:06 DEBUG "result": [ 01:13:06 DEBUG { 01:13:06 DEBUG "metric": { 01:13:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG [...] 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG 01:13:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:13:06 DEBUG }, 01:13:06 DEBUG "value": [ 01:13:06 DEBUG 1713139986, 01:13:06 DEBUG "0" 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG ] 01:13:06 DEBUG } 01:13:06 DEBUG } 01:13:24 INFO 01:13:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:13:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:13:24 INFO [loop_until]: OK (rc = 0) 01:13:24 DEBUG --- stdout --- 01:13:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8539Mi am-7d597cf7d6-8vr8f 6m 8661Mi am-7d597cf7d6-9wz9h 7m 8609Mi ds-cts-0 15m 8197Mi ds-cts-1 14m 8429Mi ds-cts-2 16m 8175Mi ds-idrepo-0 11m 4606Mi ds-idrepo-1 10m 4328Mi ds-idrepo-2 11m 4101Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1017Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 9m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 978m 3348Mi 01:13:24 DEBUG --- stderr --- 01:13:24 DEBUG 01:13:30 INFO 01:13:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:13:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:13:30 INFO [loop_until]: OK (rc = 0) 01:13:30 DEBUG --- stdout --- 01:13:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 4897Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 76m 0% 11060Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10623Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3075Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 6866Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 12472Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2936Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 66m 0% 10145Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5978Mi 5% gke-xlou-cdm-ds-0f601751-j488 68m 0% 10667Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 73m 0% 2430Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 10206Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6522Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 7630Mi 13% 01:13:30 DEBUG --- stderr --- 01:13:30 DEBUG 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 ERROR [http_cmd]: ERROR 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 ERROR --- status code --- 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:14:06 DEBUG --- http response --- 01:14:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 ERROR --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG { 01:14:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG { 01:14:06 DEBUG "data": { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "data": { 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "data": { 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG { 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "result": [ 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "metric": { 01:14:06 DEBUG { 01:14:06 DEBUG { 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG { 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG }, 01:14:06 DEBUG "metric": { 01:14:06 DEBUG }, 01:14:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:14:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:14:06 DEBUG "data": { 01:14:06 DEBUG }, 01:14:06 DEBUG }, 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "instance": "10.142.0.107:9100" 01:14:06 DEBUG "value": [ 01:14:06 DEBUG }, 01:14:06 DEBUG }, 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "value": [ 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG 01:14:06 DEBUG }, 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "result": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "metric": { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG [...] 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG [...] 01:14:06 DEBUG }, 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "data": { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG }, 01:14:06 DEBUG "result": [ 01:14:06 DEBUG 01:14:06 DEBUG }, 01:14:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG 01:14:06 DEBUG }, 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG { 01:14:06 DEBUG "instance": "10.142.0.43:9100" 01:14:06 DEBUG "value": [ 01:14:06 DEBUG }, 01:14:06 DEBUG }, 01:14:06 DEBUG [...] 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "metric": { 01:14:06 DEBUG }, 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "0" 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG 01:14:06 DEBUG "0" 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG }, 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG ] 01:14:06 DEBUG "1.0307112447181308" 01:14:06 DEBUG "0" 01:14:06 DEBUG 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "0.0005333333333330377" 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG }, 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG "value": [ 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG [...] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG "0" 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG 01:14:06 DEBUG ] 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG } 01:14:06 DEBUG }, 01:14:06 DEBUG } 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0.055" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG --- http response --- 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG { 01:14:06 DEBUG { 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 DEBUG "data": { 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG "resultType": "vector", 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG "resultType": "vector", 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG "result": [ 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG { 01:14:06 DEBUG "result": [ 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG { 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG { 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG "metric": { 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG "data": { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 DEBUG }, 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG "result": [ 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG { 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:06 DEBUG { 01:14:06 DEBUG { 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "data": { 01:14:06 DEBUG { 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:14:06 DEBUG "data": { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG }, 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "data": { 01:14:06 DEBUG { 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "result": [ 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG 01:14:06 DEBUG { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "result": [ 01:14:06 DEBUG "data": { 01:14:06 DEBUG 01:14:06 DEBUG "metric": { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 DEBUG { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG 01:14:06 DEBUG "instance": "10.142.0.107:9100" 01:14:06 DEBUG 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG }, 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "result": [ 01:14:06 DEBUG [...] 01:14:06 DEBUG }, 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG { 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG }, 01:14:06 DEBUG "metric": { 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG }, 01:14:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:14:06 DEBUG 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG }, 01:14:06 DEBUG [...] 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "3955724288" 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG ] 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG "instance": "10.142.0.43:9100" 01:14:06 DEBUG } 01:14:06 DEBUG }, 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "pod": "ds-cts-1" 01:14:06 DEBUG 01:14:06 DEBUG ] 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG }, 01:14:06 DEBUG 01:14:06 DEBUG } 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG "value": [ 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG } 01:14:06 DEBUG "13415910400" 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG ] 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG }, 01:14:06 DEBUG ] 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG } 01:14:06 DEBUG "0" 01:14:06 DEBUG "value": [ 01:14:06 DEBUG } 01:14:06 DEBUG { 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG ] 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG "0" 01:14:06 DEBUG } 01:14:06 DEBUG "data": { 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 DEBUG "result": [ 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG { 01:14:06 DEBUG } 01:14:06 DEBUG "metric": { 01:14:06 DEBUG } 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "4877.385913914474" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "instance": "10.142.0.107:9100" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "instance": "10.142.0.43:9100" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "14364.166666666664" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "ds-idrepo-0" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "instance": "10.142.0.107:9100" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "instance": "10.142.0.43:9100" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0.00033333333333333365" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "ds-idrepo-1" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "ds-cts-1" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "instance": "10.142.0.107:9100" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "instance": "10.142.0.43:9100" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "7712.433333333333" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140046" 01:14:06 INFO [http_cmd]: http status code OK 01:14:06 DEBUG --- status code --- 01:14:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:06 DEBUG --- http response --- 01:14:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:06 DEBUG ----- output ----- 01:14:06 DEBUG { 01:14:06 DEBUG "status": "success", 01:14:06 DEBUG "data": { 01:14:06 DEBUG "resultType": "vector", 01:14:06 DEBUG "result": [ 01:14:06 DEBUG { 01:14:06 DEBUG "metric": { 01:14:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG [...] 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG 01:14:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:14:06 DEBUG }, 01:14:06 DEBUG "value": [ 01:14:06 DEBUG 1713140046, 01:14:06 DEBUG "0" 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG ] 01:14:06 DEBUG } 01:14:06 DEBUG } 01:14:24 INFO 01:14:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:14:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:14:24 INFO [loop_until]: OK (rc = 0) 01:14:24 DEBUG --- stdout --- 01:14:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 7m 8610Mi ds-cts-0 16m 8198Mi ds-cts-1 15m 8431Mi ds-cts-2 16m 8174Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 10m 4329Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1017Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 997m 3998Mi 01:14:24 DEBUG --- stderr --- 01:14:24 DEBUG 01:14:30 INFO 01:14:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:14:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:14:30 INFO [loop_until]: OK (rc = 0) 01:14:30 DEBUG --- stdout --- 01:14:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4897Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 78m 0% 11039Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10616Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3082Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 6862Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 12492Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2934Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10138Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 5980Mi 5% gke-xlou-cdm-ds-0f601751-j488 67m 0% 10661Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2426Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 81m 0% 10206Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6534Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 857m 5% 7859Mi 13% 01:14:30 DEBUG --- stderr --- 01:14:30 DEBUG 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG { 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:06 DEBUG "data": { 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140106" 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140106" 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "data": { 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG "result": [ 01:15:06 DEBUG }, 01:15:06 DEBUG "result": [ 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG { 01:15:06 DEBUG "value": [ 01:15:06 DEBUG { 01:15:06 DEBUG --- http response --- 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG "metric": { 01:15:06 DEBUG 01:15:06 DEBUG "metric": { 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG "pod": "ds-cts-2" 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG }, 01:15:06 DEBUG 01:15:06 DEBUG }, 01:15:06 DEBUG { 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG "value": [ 01:15:06 DEBUG [...] 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG "data": { 01:15:06 DEBUG 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG 01:15:06 DEBUG { 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG "result": [ 01:15:06 DEBUG [...] 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG [...] 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG { 01:15:06 DEBUG 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 DEBUG "data": { 01:15:06 DEBUG }, 01:15:06 DEBUG 01:15:06 DEBUG "metric": { 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:06 DEBUG 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG 01:15:06 DEBUG "result": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG }, 01:15:06 DEBUG { 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG { 01:15:06 ERROR [http_cmd]: ERROR 01:15:06 DEBUG "0.055" 01:15:06 DEBUG "pod": "ds-idrepo-0" 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG }, 01:15:06 DEBUG "metric": { 01:15:06 ERROR --- status code --- 01:15:06 DEBUG ] 01:15:06 DEBUG }, 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG 01:15:06 DEBUG "data": { 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:15:06 DEBUG } 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG }, 01:15:06 ERROR --- http response --- 01:15:06 DEBUG ] 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG "result": [ 01:15:06 DEBUG "0" 01:15:06 DEBUG "value": [ 01:15:06 DEBUG } 01:15:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:15:06 DEBUG "0" 01:15:06 DEBUG [...] 01:15:06 DEBUG { 01:15:06 DEBUG { 01:15:06 DEBUG ] 01:15:06 DEBUG 01:15:06 DEBUG } 01:15:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:15:06 DEBUG ] 01:15:06 DEBUG 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "metric": { 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG "data": { 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG ] 01:15:06 DEBUG 01:15:06 DEBUG ] 01:15:06 DEBUG 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG }, 01:15:06 DEBUG } 01:15:06 DEBUG [...] 01:15:06 DEBUG } 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG "result": [ 01:15:06 DEBUG "value": [ 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG } 01:15:06 DEBUG }, 01:15:06 DEBUG { 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "metric": { 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG "0" 01:15:06 DEBUG }, 01:15:06 DEBUG [...] 01:15:06 DEBUG }, 01:15:06 DEBUG ] 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG ] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "0" 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-idrepo-0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG [...] 01:15:06 DEBUG }, 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG ] 01:15:06 DEBUG 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG } 01:15:06 DEBUG 01:15:06 DEBUG "0" 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG } 01:15:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:15:06 DEBUG ] 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG }, 01:15:06 DEBUG } 01:15:06 DEBUG "value": [ 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG ] 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG } 01:15:06 DEBUG "0.9338122869253555" 01:15:06 DEBUG { 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG } 01:15:06 DEBUG "data": { 01:15:06 DEBUG ] 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG } 01:15:06 DEBUG "result": [ 01:15:06 DEBUG } 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "resultType": "vector", 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 DEBUG "result": [ 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 DEBUG "instance": "10.142.0.107:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG "instance": "10.142.0.43:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG "0.0002666666666669926" 01:15:06 DEBUG ] 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "ds-cts-2" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0.0034582955289453885" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "ds-cts-2" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "314.4137636160033" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-idrepo-0" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- http response --- 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG { 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG "data": { 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "data": { 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG "result": [ 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG { 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG { 01:15:06 DEBUG "result": [ 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "metric": { 01:15:06 DEBUG { 01:15:06 DEBUG "data": { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "data": { 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "instance": "10.142.0.107:9100" 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG }, 01:15:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:15:06 DEBUG "result": [ 01:15:06 DEBUG }, 01:15:06 DEBUG "result": [ 01:15:06 DEBUG "value": [ 01:15:06 DEBUG }, 01:15:06 DEBUG { 01:15:06 DEBUG "value": [ 01:15:06 DEBUG { 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 DEBUG "metric": { 01:15:06 DEBUG 01:15:06 DEBUG "metric": { 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG }, 01:15:06 DEBUG 01:15:06 DEBUG }, 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 DEBUG [...] 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-cts-1" 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG "instance": "10.142.0.43:9100" 01:15:06 DEBUG [...] 01:15:06 DEBUG }, 01:15:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:15:06 DEBUG 01:15:06 DEBUG }, 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 DEBUG }, 01:15:06 DEBUG 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG 01:15:06 DEBUG "0" 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG "13416537088" 01:15:06 DEBUG "pod": "ds-idrepo-0" 01:15:06 DEBUG ] 01:15:06 DEBUG "4272971776" 01:15:06 DEBUG }, 01:15:06 DEBUG ] 01:15:06 DEBUG }, 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG "value": [ 01:15:06 DEBUG } 01:15:06 DEBUG "value": [ 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG ] 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG "0" 01:15:06 DEBUG } 01:15:06 DEBUG "0" 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "instance": "10.142.0.107:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "instance": "10.142.0.43:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "14242.566666666666" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-idrepo-0" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "ds-cts-2" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "34899.92776137636" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "instance": "10.142.0.107:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "instance": "10.142.0.43:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "ds-idrepo-1" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "ds-cts-1" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "instance": "10.142.0.107:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "instance": "10.142.0.43:9100" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "7927.7" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140106" 01:15:06 INFO [http_cmd]: http status code OK 01:15:06 DEBUG --- status code --- 01:15:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:06 DEBUG --- http response --- 01:15:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:06 DEBUG ----- output ----- 01:15:06 DEBUG { 01:15:06 DEBUG "status": "success", 01:15:06 DEBUG "data": { 01:15:06 DEBUG "resultType": "vector", 01:15:06 DEBUG "result": [ 01:15:06 DEBUG { 01:15:06 DEBUG "metric": { 01:15:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG [...] 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG 01:15:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:15:06 DEBUG }, 01:15:06 DEBUG "value": [ 01:15:06 DEBUG 1713140106, 01:15:06 DEBUG "0" 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG ] 01:15:06 DEBUG } 01:15:06 DEBUG } 01:15:24 INFO 01:15:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:15:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:15:24 INFO [loop_until]: OK (rc = 0) 01:15:24 DEBUG --- stdout --- 01:15:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 7m 8610Mi ds-cts-0 17m 8198Mi ds-cts-1 14m 8431Mi ds-cts-2 14m 8174Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 11m 4328Mi ds-idrepo-2 11m 4098Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 566m 4238Mi 01:15:24 DEBUG --- stderr --- 01:15:24 DEBUG 01:15:30 INFO 01:15:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:15:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:15:30 INFO [loop_until]: OK (rc = 0) 01:15:30 DEBUG --- stdout --- 01:15:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4885Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 11057Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10612Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3095Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 6869Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 12488Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2949Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 57m 0% 10120Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5974Mi 5% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10658Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2420Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 10195Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 6529Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1144m 7% 8359Mi 14% 01:15:30 DEBUG --- stderr --- 01:15:30 DEBUG 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 ERROR [http_cmd]: ERROR 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG --- status code --- 01:16:06 ERROR --- status code --- 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:16:06 DEBUG { 01:16:06 DEBUG --- http response --- 01:16:06 ERROR --- http response --- 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 01:16:06 DEBUG "data": { 01:16:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG "resultType": "vector", 01:16:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:16:06 DEBUG { 01:16:06 DEBUG "result": [ 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG "status": "success", 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG { 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG "data": { 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG "metric": { 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG "result": [ 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG }, 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG { 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:16:06 DEBUG "value": [ 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG "metric": { 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG ----- output ----- 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG { 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG 01:16:06 DEBUG { 01:16:06 DEBUG }, 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "data": { 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [...] 01:16:06 DEBUG "data": { 01:16:06 DEBUG 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG "result": [ 01:16:06 DEBUG 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG "result": [ 01:16:06 DEBUG 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG { 01:16:06 DEBUG 01:16:06 DEBUG { 01:16:06 DEBUG { 01:16:06 DEBUG [...] 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "metric": { 01:16:06 DEBUG 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "data": { 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG }, 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG }, 01:16:06 DEBUG 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "result": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "pod": "ds-cts-1" 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "0.055" 01:16:06 DEBUG { 01:16:06 DEBUG 01:16:06 DEBUG }, 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG ] 01:16:06 DEBUG "metric": { 01:16:06 DEBUG 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG } 01:16:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:16:06 DEBUG 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG [...] 01:16:06 DEBUG ] 01:16:06 DEBUG }, 01:16:06 DEBUG [...] 01:16:06 DEBUG "0" 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG } 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG ] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG } 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG } 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG ] 01:16:06 DEBUG 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG } 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG }, 01:16:06 DEBUG }, 01:16:06 DEBUG [...] 01:16:06 DEBUG }, 01:16:06 DEBUG } 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG "0" 01:16:06 DEBUG 01:16:06 DEBUG "0" 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG ] 01:16:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:16:06 DEBUG ] 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG }, 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG ] 01:16:06 DEBUG "value": [ 01:16:06 DEBUG ] 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG "1.0863167327582075" 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "data": { 01:16:06 DEBUG "instance": "10.142.0.107:9100" 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG }, 01:16:06 DEBUG "result": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG { 01:16:06 DEBUG 01:16:06 DEBUG "metric": { 01:16:06 DEBUG 01:16:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:16:06 DEBUG 01:16:06 DEBUG }, 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "instance": "10.142.0.43:9100" 01:16:06 DEBUG 01:16:06 DEBUG }, 01:16:06 DEBUG [...] 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG 01:16:06 DEBUG "0.00036666666666652263" 01:16:06 DEBUG 01:16:06 DEBUG ] 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG } 01:16:06 DEBUG }, 01:16:06 DEBUG ] 01:16:06 DEBUG "value": [ 01:16:06 DEBUG } 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG } 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG }, 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG }, 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-cts-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "571.6837874050916" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-cts-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "4890935296" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG } 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG } 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "instance": "10.142.0.107:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "instance": "10.142.0.43:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "13417088000" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "instance": "10.142.0.107:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "instance": "10.142.0.43:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "14767.999999999998" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-idrepo-0" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "instance": "10.142.0.107:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "instance": "10.142.0.43:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "ds-idrepo-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "ds-cts-1" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "instance": "10.142.0.107:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "instance": "10.142.0.43:9100" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "7868.1" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140166" 01:16:06 INFO [http_cmd]: http status code OK 01:16:06 DEBUG --- status code --- 01:16:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:06 DEBUG --- http response --- 01:16:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:06 DEBUG ----- output ----- 01:16:06 DEBUG { 01:16:06 DEBUG "status": "success", 01:16:06 DEBUG "data": { 01:16:06 DEBUG "resultType": "vector", 01:16:06 DEBUG "result": [ 01:16:06 DEBUG { 01:16:06 DEBUG "metric": { 01:16:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG [...] 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG 01:16:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:16:06 DEBUG }, 01:16:06 DEBUG "value": [ 01:16:06 DEBUG 1713140166, 01:16:06 DEBUG "0" 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG ] 01:16:06 DEBUG } 01:16:06 DEBUG } 01:16:24 INFO 01:16:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:16:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:16:24 INFO [loop_until]: OK (rc = 0) 01:16:24 DEBUG --- stdout --- 01:16:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8539Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 6m 8610Mi ds-cts-0 13m 8197Mi ds-cts-1 12m 8430Mi ds-cts-2 12m 8174Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 10m 4328Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1002m 4770Mi 01:16:24 DEBUG --- stderr --- 01:16:24 DEBUG 01:16:30 INFO 01:16:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:16:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:16:30 INFO [loop_until]: OK (rc = 0) 01:16:30 DEBUG --- stdout --- 01:16:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4886Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 76m 0% 11054Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10598Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3104Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 6859Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 12484Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2947Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10138Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 66m 0% 5976Mi 5% gke-xlou-cdm-ds-0f601751-j488 65m 0% 10648Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2426Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 89m 0% 10195Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 6533Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1086m 6% 9118Mi 15% 01:16:30 DEBUG --- stderr --- 01:16:30 DEBUG 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 ERROR [http_cmd]: ERROR 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 ERROR --- status code --- 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 ERROR --- http response --- 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:17:06 DEBUG { 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:06 DEBUG }, 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:17:06 DEBUG 01:17:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:06 DEBUG }, 01:17:06 DEBUG "pod": "ds-idrepo-0" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "1.0357862169628516" 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG 01:17:06 DEBUG { 01:17:06 DEBUG "pod": "ds-idrepo-0" 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG }, 01:17:06 DEBUG "data": { 01:17:06 DEBUG "value": [ 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "result": [ 01:17:06 DEBUG "0" 01:17:06 DEBUG { 01:17:06 DEBUG ] 01:17:06 DEBUG "metric": { 01:17:06 DEBUG } 01:17:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:06 DEBUG ] 01:17:06 DEBUG }, 01:17:06 DEBUG } 01:17:06 DEBUG "value": [ 01:17:06 DEBUG } 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "lodemon-5d8fc565dd-9g64c" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "ds-idrepo-0" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0.055" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "lodemon-5d8fc565dd-9g64c" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "idm-b85f9d978-rl76f" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "instance": "10.142.0.107:9100" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "instance": "10.142.0.43:9100" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0.00043333333333350765" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "ds-idrepo-0" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "data": { 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140226" 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "instance": "10.142.0.107:9100" 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:17:06 DEBUG }, 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:06 DEBUG 01:17:06 DEBUG "instance": "10.142.0.43:9100" 01:17:06 DEBUG "pod": "ds-cts-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:17:06 DEBUG }, 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "13417794560" 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG "5557669888" 01:17:06 DEBUG ] 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "instance": "10.142.0.107:9100" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "instance": "10.142.0.43:9100" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "14960.233333333332" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "ds-idrepo-0" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "idm-b85f9d978-rl76f" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "instance": "10.142.0.107:9100" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "instance": "10.142.0.43:9100" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0.00033333333333333365" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140226" 01:17:06 INFO [http_cmd]: http status code OK 01:17:06 DEBUG --- status code --- 01:17:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:06 DEBUG --- http response --- 01:17:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:17:06 DEBUG ----- output ----- 01:17:06 DEBUG { 01:17:06 DEBUG "status": "success", 01:17:06 DEBUG "data": { 01:17:06 DEBUG "resultType": "vector", 01:17:06 DEBUG "result": [ 01:17:06 DEBUG { 01:17:06 DEBUG "metric": { 01:17:06 DEBUG "pod": "ds-idrepo-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG [...] 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG 01:17:06 DEBUG "pod": "ds-cts-1" 01:17:06 DEBUG }, 01:17:06 DEBUG "value": [ 01:17:06 DEBUG 1713140226, 01:17:06 DEBUG "0" 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG ] 01:17:06 DEBUG } 01:17:06 DEBUG } 01:17:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140226" 01:17:07 INFO [http_cmd]: http status code OK 01:17:07 DEBUG --- status code --- 01:17:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:07 DEBUG --- http response --- 01:17:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:07 DEBUG ----- output ----- 01:17:07 DEBUG { 01:17:07 DEBUG "status": "success", 01:17:07 DEBUG "data": { 01:17:07 DEBUG "resultType": "vector", 01:17:07 DEBUG "result": [ 01:17:07 DEBUG { 01:17:07 DEBUG "metric": { 01:17:07 DEBUG "instance": "10.142.0.107:9100" 01:17:07 DEBUG }, 01:17:07 DEBUG "value": [ 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG [...] 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG "instance": "10.142.0.43:9100" 01:17:07 DEBUG }, 01:17:07 DEBUG "value": [ 01:17:07 DEBUG 1713140226, 01:17:07 DEBUG "14830" 01:17:07 DEBUG ] 01:17:07 DEBUG } 01:17:07 DEBUG ] 01:17:07 DEBUG } 01:17:07 DEBUG } 01:17:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140227" 01:17:07 INFO [http_cmd]: http status code OK 01:17:07 DEBUG --- status code --- 01:17:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:07 DEBUG --- http response --- 01:17:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:07 DEBUG ----- output ----- 01:17:07 DEBUG { 01:17:07 DEBUG "status": "success", 01:17:07 DEBUG "data": { 01:17:07 DEBUG "resultType": "vector", 01:17:07 DEBUG "result": [ 01:17:07 DEBUG { 01:17:07 DEBUG "metric": { 01:17:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:17:07 DEBUG }, 01:17:07 DEBUG "value": [ 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG [...] 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG 01:17:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:17:07 DEBUG }, 01:17:07 DEBUG "value": [ 01:17:07 DEBUG 1713140227, 01:17:07 DEBUG "0" 01:17:07 DEBUG ] 01:17:07 DEBUG } 01:17:07 DEBUG ] 01:17:07 DEBUG } 01:17:07 DEBUG } 01:17:24 INFO 01:17:24 INFO [loop_until]: kubectl --namespace=xlou top pods 01:17:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:17:24 INFO [loop_until]: OK (rc = 0) 01:17:24 DEBUG --- stdout --- 01:17:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 6m 8609Mi ds-cts-0 12m 8196Mi ds-cts-1 11m 8430Mi ds-cts-2 12m 8174Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 9m 4328Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1016Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 991m 5568Mi 01:17:24 DEBUG --- stderr --- 01:17:24 DEBUG 01:17:30 INFO 01:17:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:17:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:17:30 INFO [loop_until]: OK (rc = 0) 01:17:30 DEBUG --- stdout --- 01:17:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4888Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 11047Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 10609Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3093Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6858Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 12467Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2939Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 63m 0% 10136Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 69m 0% 5969Mi 5% gke-xlou-cdm-ds-0f601751-j488 60m 0% 10657Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2429Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 82m 0% 10198Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6545Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1046m 6% 9850Mi 16% 01:17:30 DEBUG --- stderr --- 01:17:30 DEBUG 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140286" 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 ERROR [http_cmd]: ERROR 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 ERROR --- status code --- 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 ERROR --- http response --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG { 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG "data": { 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "data": { 01:18:06 DEBUG { 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 DEBUG "result": [ 01:18:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG { 01:18:06 DEBUG }, 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "data": { 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "value": [ 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "ds-idrepo-1" 01:18:06 DEBUG "pod": "ds-cts-2" 01:18:06 DEBUG }, 01:18:06 DEBUG "pod": "ds-idrepo-1" 01:18:06 DEBUG 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG "pod": "ds-idrepo-1" 01:18:06 DEBUG }, 01:18:06 DEBUG "result": [ 01:18:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:18:06 DEBUG }, 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG }, 01:18:06 DEBUG 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG { 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG "value": [ 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:18:06 DEBUG 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG 01:18:06 DEBUG "instance": "10.142.0.107:9100" 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG { 01:18:06 DEBUG 01:18:06 DEBUG }, 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG [...] 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG "data": { 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG }, 01:18:06 DEBUG 01:18:06 DEBUG "result": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG { 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG "pod": "ds-idrepo-0" 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG }, 01:18:06 DEBUG "pod": "ds-idrepo-0" 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "pod": "ds-idrepo-0" 01:18:06 DEBUG "metric": { 01:18:06 DEBUG }, 01:18:06 DEBUG 01:18:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:18:06 DEBUG }, 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG }, 01:18:06 DEBUG "0" 01:18:06 DEBUG }, 01:18:06 DEBUG "pod": "ds-cts-2" 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG ] 01:18:06 DEBUG "value": [ 01:18:06 DEBUG }, 01:18:06 DEBUG 1713140286, 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG ] 01:18:06 DEBUG "value": [ 01:18:06 DEBUG } 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG "0" 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG "instance": "10.142.0.43:9100" 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG "0.00460943223900493" 01:18:06 DEBUG } 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG ] 01:18:06 DEBUG "0" 01:18:06 DEBUG 01:18:06 DEBUG ] 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG }, 01:18:06 DEBUG "1.0284188847946847" 01:18:06 DEBUG ] 01:18:06 DEBUG ] 01:18:06 DEBUG ] 01:18:06 DEBUG "0" 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG 01:18:06 DEBUG } 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG "value": [ 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG 01:18:06 DEBUG ] 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG [...] 01:18:06 DEBUG } 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG "0.00043333333333350765" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG 01:18:06 DEBUG } 01:18:06 DEBUG { 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG 01:18:06 DEBUG "status": "success", 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 DEBUG 01:18:06 DEBUG "data": { 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG } 01:18:06 DEBUG }, 01:18:06 DEBUG "result": [ 01:18:06 DEBUG } 01:18:06 DEBUG "value": [ 01:18:06 DEBUG { 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "0" 01:18:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:06 DEBUG ] 01:18:06 DEBUG }, 01:18:06 DEBUG } 01:18:06 DEBUG "value": [ 01:18:06 DEBUG ] 01:18:06 DEBUG 01:18:06 DEBUG } 01:18:06 DEBUG 01:18:06 DEBUG } 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0.055" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "ds-cts-2" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "338.17566137566143" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "ds-idrepo-1" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "ds-idrepo-0" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140286" 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "6356422656" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "ds-idrepo-1" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "ds-cts-1" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG { 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG "data": { 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "data": { 01:18:06 DEBUG "instance": "10.142.0.107:9100" 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG }, 01:18:06 DEBUG "result": [ 01:18:06 DEBUG "value": [ 01:18:06 DEBUG { 01:18:06 DEBUG 01:18:06 DEBUG "metric": { 01:18:06 DEBUG 01:18:06 DEBUG "instance": "10.142.0.107:9100" 01:18:06 DEBUG 01:18:06 DEBUG }, 01:18:06 DEBUG [...] 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "instance": "10.142.0.43:9100" 01:18:06 DEBUG [...] 01:18:06 DEBUG }, 01:18:06 DEBUG 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG 01:18:06 DEBUG "14337.933333333332" 01:18:06 DEBUG "instance": "10.142.0.43:9100" 01:18:06 DEBUG ] 01:18:06 DEBUG }, 01:18:06 DEBUG } 01:18:06 DEBUG "value": [ 01:18:06 DEBUG ] 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG } 01:18:06 DEBUG "13418439680" 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140286" 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "ds-idrepo-1" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "ds-idrepo-0" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "pod": "ds-cts-2" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "11610.697707231042" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140286" 01:18:06 INFO [http_cmd]: http status code OK 01:18:06 DEBUG --- status code --- 01:18:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:06 DEBUG --- http response --- 01:18:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:06 DEBUG ----- output ----- 01:18:06 DEBUG { 01:18:06 DEBUG "status": "success", 01:18:06 DEBUG "data": { 01:18:06 DEBUG "resultType": "vector", 01:18:06 DEBUG "result": [ 01:18:06 DEBUG { 01:18:06 DEBUG "metric": { 01:18:06 DEBUG "instance": "10.142.0.107:9100" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG [...] 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG 01:18:06 DEBUG "instance": "10.142.0.43:9100" 01:18:06 DEBUG }, 01:18:06 DEBUG "value": [ 01:18:06 DEBUG 1713140286, 01:18:06 DEBUG "0.0006666666666666673" 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG ] 01:18:06 DEBUG } 01:18:06 DEBUG } 01:18:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140287" 01:18:07 INFO [http_cmd]: http status code OK 01:18:07 DEBUG --- status code --- 01:18:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:07 DEBUG --- http response --- 01:18:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:18:07 DEBUG ----- output ----- 01:18:07 DEBUG { 01:18:07 DEBUG "status": "success", 01:18:07 DEBUG "data": { 01:18:07 DEBUG "resultType": "vector", 01:18:07 DEBUG "result": [ 01:18:07 DEBUG { 01:18:07 DEBUG "metric": { 01:18:07 DEBUG "pod": "ds-idrepo-1" 01:18:07 DEBUG }, 01:18:07 DEBUG "value": [ 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG [...] 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG "pod": "ds-cts-1" 01:18:07 DEBUG }, 01:18:07 DEBUG "value": [ 01:18:07 DEBUG 1713140287, 01:18:07 DEBUG "0" 01:18:07 DEBUG ] 01:18:07 DEBUG } 01:18:07 DEBUG ] 01:18:07 DEBUG } 01:18:07 DEBUG } 01:18:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140287" 01:18:07 INFO [http_cmd]: http status code OK 01:18:07 DEBUG --- status code --- 01:18:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:07 DEBUG --- http response --- 01:18:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:07 DEBUG ----- output ----- 01:18:07 DEBUG { 01:18:07 DEBUG "status": "success", 01:18:07 DEBUG "data": { 01:18:07 DEBUG "resultType": "vector", 01:18:07 DEBUG "result": [ 01:18:07 DEBUG { 01:18:07 DEBUG "metric": { 01:18:07 DEBUG "instance": "10.142.0.107:9100" 01:18:07 DEBUG }, 01:18:07 DEBUG "value": [ 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG [...] 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG "instance": "10.142.0.43:9100" 01:18:07 DEBUG }, 01:18:07 DEBUG "value": [ 01:18:07 DEBUG 1713140287, 01:18:07 DEBUG "8992.766666666668" 01:18:07 DEBUG ] 01:18:07 DEBUG } 01:18:07 DEBUG ] 01:18:07 DEBUG } 01:18:07 DEBUG } 01:18:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140287" 01:18:07 INFO [http_cmd]: http status code OK 01:18:07 DEBUG --- status code --- 01:18:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:07 DEBUG --- http response --- 01:18:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:07 DEBUG ----- output ----- 01:18:07 DEBUG { 01:18:07 DEBUG "status": "success", 01:18:07 DEBUG "data": { 01:18:07 DEBUG "resultType": "vector", 01:18:07 DEBUG "result": [ 01:18:07 DEBUG { 01:18:07 DEBUG "metric": { 01:18:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:18:07 DEBUG }, 01:18:07 DEBUG "value": [ 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG [...] 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG 01:18:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:18:07 DEBUG }, 01:18:07 DEBUG "value": [ 01:18:07 DEBUG 1713140287, 01:18:07 DEBUG "0" 01:18:07 DEBUG ] 01:18:07 DEBUG } 01:18:07 DEBUG ] 01:18:07 DEBUG } 01:18:07 DEBUG } 01:18:25 INFO 01:18:25 INFO [loop_until]: kubectl --namespace=xlou top pods 01:18:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:18:25 INFO [loop_until]: OK (rc = 0) 01:18:25 DEBUG --- stdout --- 01:18:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 7m 8609Mi ds-cts-0 13m 8196Mi ds-cts-1 10m 8430Mi ds-cts-2 13m 8173Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 10m 4327Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 993m 6295Mi 01:18:25 DEBUG --- stderr --- 01:18:25 DEBUG 01:18:30 INFO 01:18:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:18:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:18:30 INFO [loop_until]: OK (rc = 0) 01:18:30 DEBUG --- stdout --- 01:18:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4886Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 11049Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 10621Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 93m 0% 3093Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6849Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 12470Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2950Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10143Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 68m 0% 5974Mi 5% gke-xlou-cdm-ds-0f601751-j488 65m 0% 10676Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2429Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 10217Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 6554Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1022m 6% 10245Mi 17% 01:18:30 DEBUG --- stderr --- 01:18:30 DEBUG 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG }, 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG "value": [ 01:19:06 DEBUG { 01:19:06 DEBUG 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG 01:19:06 DEBUG "data": { 01:19:06 DEBUG 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG [...] 01:19:06 DEBUG "result": [ 01:19:06 DEBUG 01:19:06 DEBUG { 01:19:06 DEBUG 01:19:06 DEBUG "metric": { 01:19:06 DEBUG 01:19:06 DEBUG "pod": "ds-cts-2" 01:19:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:06 DEBUG }, 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG 01:19:06 DEBUG "0" 01:19:06 DEBUG 01:19:06 DEBUG ] 01:19:06 DEBUG [...] 01:19:06 DEBUG } 01:19:06 DEBUG 01:19:06 DEBUG ] 01:19:06 DEBUG 01:19:06 DEBUG } 01:19:06 DEBUG 01:19:06 DEBUG } 01:19:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 DEBUG --- status code --- 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 ERROR [http_cmd]: ERROR 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 ERROR --- status code --- 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- status code --- 01:19:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- status code --- 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 ERROR --- http response --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG ----- output ----- 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG { 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG ----- output ----- 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:06 DEBUG { 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG { 01:19:06 DEBUG { 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 01:19:06 DEBUG { 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG { 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG "data": { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG { 01:19:06 DEBUG "data": { 01:19:06 DEBUG { 01:19:06 DEBUG "data": { 01:19:06 DEBUG "data": { 01:19:06 DEBUG { 01:19:06 DEBUG "data": { 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "data": { 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG { 01:19:06 DEBUG "data": { 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "data": { 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "data": { 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG { 01:19:06 DEBUG { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG { 01:19:06 DEBUG { 01:19:06 DEBUG "data": { 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "pod": "ds-idrepo-1" 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:06 DEBUG { 01:19:06 DEBUG "instance": "10.142.0.107:9100" 01:19:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:06 DEBUG { 01:19:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:06 DEBUG }, 01:19:06 DEBUG "data": { 01:19:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "data": { 01:19:06 DEBUG }, 01:19:06 DEBUG "metric": { 01:19:06 DEBUG }, 01:19:06 DEBUG }, 01:19:06 DEBUG "metric": { 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG }, 01:19:06 DEBUG { 01:19:06 DEBUG "pod": "ds-idrepo-1" 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "pod": "ds-idrepo-1" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "pod": "ds-cts-2" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "metric": { 01:19:06 DEBUG }, 01:19:06 DEBUG "result": [ 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG { 01:19:06 DEBUG 01:19:06 DEBUG "pod": "ds-idrepo-1" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG { 01:19:06 DEBUG 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "metric": { 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG "metric": { 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG "pod": "ds-cts-2" 01:19:06 DEBUG 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "pod": "ds-idrepo-0" 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:06 DEBUG 01:19:06 DEBUG "instance": "10.142.0.43:9100" 01:19:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:06 DEBUG 01:19:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG [...] 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG "pod": "ds-idrepo-0" 01:19:06 DEBUG [...] 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "pod": "ds-idrepo-0" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG }, 01:19:06 DEBUG 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG }, 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG }, 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0" 01:19:06 DEBUG 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "pod": "ds-idrepo-0" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG "0.055" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "0.00033333333333350387" 01:19:06 DEBUG "0" 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "0" 01:19:06 DEBUG ] 01:19:06 DEBUG 01:19:06 DEBUG "0" 01:19:06 DEBUG }, 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG 01:19:06 DEBUG ] 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 01:19:06 DEBUG ] 01:19:06 DEBUG "value": [ 01:19:06 DEBUG "0" 01:19:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:19:06 DEBUG } 01:19:06 DEBUG "0" 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG "0" 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG }, 01:19:06 DEBUG } 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG ] 01:19:06 DEBUG }, 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG "value": [ 01:19:06 DEBUG ] 01:19:06 DEBUG "0" 01:19:06 DEBUG } 01:19:06 DEBUG "value": [ 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG "0" 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG "0.8702757400121488" 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140346" 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG { 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "instance": "10.142.0.107:9100" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG [...] 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG 01:19:06 DEBUG { 01:19:06 DEBUG 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG 01:19:06 DEBUG "data": { 01:19:06 DEBUG "instance": "10.142.0.43:9100" 01:19:06 DEBUG "resultType": "vector", 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 DEBUG }, 01:19:06 DEBUG "result": [ 01:19:06 DEBUG "value": [ 01:19:06 DEBUG { 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "13419203584" 01:19:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:19:06 DEBUG ] 01:19:06 DEBUG }, 01:19:06 DEBUG } 01:19:06 DEBUG "value": [ 01:19:06 DEBUG ] 01:19:06 DEBUG 01:19:06 DEBUG } 01:19:06 DEBUG 01:19:06 DEBUG } 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "6779318272" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "pod": "ds-idrepo-1" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "pod": "ds-cts-1" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "instance": "10.142.0.107:9100" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "instance": "10.142.0.43:9100" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "14887.033333333333" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "pod": "ds-idrepo-1" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "pod": "ds-idrepo-0" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "pod": "ds-cts-2" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140346" 01:19:06 INFO [http_cmd]: http status code OK 01:19:06 DEBUG --- status code --- 01:19:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:06 DEBUG --- http response --- 01:19:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:06 DEBUG ----- output ----- 01:19:06 DEBUG { 01:19:06 DEBUG "status": "success", 01:19:06 DEBUG "data": { 01:19:06 DEBUG "resultType": "vector", 01:19:06 DEBUG "result": [ 01:19:06 DEBUG { 01:19:06 DEBUG "metric": { 01:19:06 DEBUG "instance": "10.142.0.107:9100" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG [...] 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG 01:19:06 DEBUG "instance": "10.142.0.43:9100" 01:19:06 DEBUG }, 01:19:06 DEBUG "value": [ 01:19:06 DEBUG 1713140346, 01:19:06 DEBUG "0.00033333333333333175" 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG ] 01:19:06 DEBUG } 01:19:06 DEBUG } 01:19:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140347" 01:19:07 INFO [http_cmd]: http status code OK 01:19:07 DEBUG --- status code --- 01:19:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:07 DEBUG --- http response --- 01:19:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:19:07 DEBUG ----- output ----- 01:19:07 DEBUG { 01:19:07 DEBUG "status": "success", 01:19:07 DEBUG "data": { 01:19:07 DEBUG "resultType": "vector", 01:19:07 DEBUG "result": [ 01:19:07 DEBUG { 01:19:07 DEBUG "metric": { 01:19:07 DEBUG "pod": "ds-idrepo-1" 01:19:07 DEBUG }, 01:19:07 DEBUG "value": [ 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG [...] 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG "pod": "ds-cts-1" 01:19:07 DEBUG }, 01:19:07 DEBUG "value": [ 01:19:07 DEBUG 1713140347, 01:19:07 DEBUG "0" 01:19:07 DEBUG ] 01:19:07 DEBUG } 01:19:07 DEBUG ] 01:19:07 DEBUG } 01:19:07 DEBUG } 01:19:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140347" 01:19:07 INFO [http_cmd]: http status code OK 01:19:07 DEBUG --- status code --- 01:19:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:07 DEBUG --- http response --- 01:19:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:07 DEBUG ----- output ----- 01:19:07 DEBUG { 01:19:07 DEBUG "status": "success", 01:19:07 DEBUG "data": { 01:19:07 DEBUG "resultType": "vector", 01:19:07 DEBUG "result": [ 01:19:07 DEBUG { 01:19:07 DEBUG "metric": { 01:19:07 DEBUG "instance": "10.142.0.107:9100" 01:19:07 DEBUG }, 01:19:07 DEBUG "value": [ 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG [...] 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG "instance": "10.142.0.43:9100" 01:19:07 DEBUG }, 01:19:07 DEBUG "value": [ 01:19:07 DEBUG 1713140347, 01:19:07 DEBUG "12389.7" 01:19:07 DEBUG ] 01:19:07 DEBUG } 01:19:07 DEBUG ] 01:19:07 DEBUG } 01:19:07 DEBUG } 01:19:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140347" 01:19:07 INFO [http_cmd]: http status code OK 01:19:07 DEBUG --- status code --- 01:19:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:07 DEBUG --- http response --- 01:19:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:07 DEBUG ----- output ----- 01:19:07 DEBUG { 01:19:07 DEBUG "status": "success", 01:19:07 DEBUG "data": { 01:19:07 DEBUG "resultType": "vector", 01:19:07 DEBUG "result": [ 01:19:07 DEBUG { 01:19:07 DEBUG "metric": { 01:19:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:19:07 DEBUG }, 01:19:07 DEBUG "value": [ 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG [...] 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG 01:19:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:19:07 DEBUG }, 01:19:07 DEBUG "value": [ 01:19:07 DEBUG 1713140347, 01:19:07 DEBUG "0" 01:19:07 DEBUG ] 01:19:07 DEBUG } 01:19:07 DEBUG ] 01:19:07 DEBUG } 01:19:07 DEBUG } 01:19:25 INFO 01:19:25 INFO [loop_until]: kubectl --namespace=xlou top pods 01:19:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:19:25 INFO [loop_until]: OK (rc = 0) 01:19:25 DEBUG --- stdout --- 01:19:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8539Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 7m 8610Mi ds-cts-0 12m 8196Mi ds-cts-1 11m 8431Mi ds-cts-2 13m 8173Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 11m 4328Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1016Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7m 6376Mi 01:19:25 DEBUG --- stderr --- 01:19:25 DEBUG 01:19:30 INFO 01:19:30 INFO [loop_until]: kubectl --namespace=xlou top node 01:19:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:19:30 INFO [loop_until]: OK (rc = 0) 01:19:30 DEBUG --- stdout --- 01:19:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4891Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 74m 0% 11049Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10615Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3090Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6851Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 12474Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2956Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10137Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 66m 0% 5975Mi 5% gke-xlou-cdm-ds-0f601751-j488 63m 0% 10676Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2418Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 10207Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 74m 0% 6530Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1071m 6% 10520Mi 17% 01:19:30 DEBUG --- stderr --- 01:19:30 DEBUG 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140406" 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0.0032704392512326037" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 ERROR [http_cmd]: ERROR 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 ERROR --- status code --- 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG --- http response --- 01:20:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 ERROR --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "data": { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "ds-idrepo-1" 01:20:06 DEBUG "data": { 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:20:06 DEBUG }, 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "result": [ 01:20:06 DEBUG "result": [ 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "result": [ 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:20:06 DEBUG "pod": "ds-idrepo-1" 01:20:06 DEBUG }, 01:20:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "ds-idrepo-0" 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG [...] 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG }, 01:20:06 DEBUG [...] 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "419.5241004910535" 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:20:06 DEBUG "pod": "ds-idrepo-0" 01:20:06 DEBUG }, 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0" 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "1.0243314346702521" 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG "0.055" 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG "0" 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140406" 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "ds-idrepo-1" 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:06 DEBUG "pod": "ds-idrepo-1" 01:20:06 DEBUG }, 01:20:06 DEBUG "instance": "10.142.0.107:9100" 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "ds-idrepo-0" 01:20:06 DEBUG 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG "pod": "ds-idrepo-0" 01:20:06 DEBUG }, 01:20:06 DEBUG "instance": "10.142.0.43:9100" 01:20:06 DEBUG }, 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0" 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0" 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG "0.00020000000000000757" 01:20:06 DEBUG ] 01:20:06 DEBUG ] 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG ] 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG } 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "instance": "10.142.0.107:9100" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "instance": "10.142.0.43:9100" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "14655.099999999999" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140406" 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "instance": "10.142.0.107:9100" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "instance": "10.142.0.43:9100" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "13419713536" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "7074918400" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "ds-idrepo-1" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "ds-cts-1" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140406" 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "ds-idrepo-1" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "ds-idrepo-0" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "0" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:06 INFO [http_cmd]: http status code OK 01:20:06 DEBUG --- status code --- 01:20:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:06 DEBUG --- http response --- 01:20:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:20:06 DEBUG ----- output ----- 01:20:06 DEBUG { 01:20:06 DEBUG "status": "success", 01:20:06 DEBUG "data": { 01:20:06 DEBUG "resultType": "vector", 01:20:06 DEBUG "result": [ 01:20:06 DEBUG { 01:20:06 DEBUG "metric": { 01:20:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG [...] 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG 01:20:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:06 DEBUG }, 01:20:06 DEBUG "value": [ 01:20:06 DEBUG 1713140406, 01:20:06 DEBUG "231196375.58781564" 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG ] 01:20:06 DEBUG } 01:20:06 DEBUG } 01:20:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140407" 01:20:07 INFO [http_cmd]: http status code OK 01:20:07 DEBUG --- status code --- 01:20:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:07 DEBUG --- http response --- 01:20:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:07 DEBUG ----- output ----- 01:20:07 DEBUG { 01:20:07 DEBUG "status": "success", 01:20:07 DEBUG "data": { 01:20:07 DEBUG "resultType": "vector", 01:20:07 DEBUG "result": [ 01:20:07 DEBUG { 01:20:07 DEBUG "metric": { 01:20:07 DEBUG "instance": "10.142.0.107:9100" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG [...] 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG "instance": "10.142.0.43:9100" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 1713140407, 01:20:07 DEBUG "0" 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG } 01:20:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140407" 01:20:07 INFO [http_cmd]: http status code OK 01:20:07 DEBUG --- status code --- 01:20:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:07 DEBUG --- http response --- 01:20:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:20:07 DEBUG ----- output ----- 01:20:07 DEBUG { 01:20:07 DEBUG "status": "success", 01:20:07 DEBUG "data": { 01:20:07 DEBUG "resultType": "vector", 01:20:07 DEBUG "result": [ 01:20:07 DEBUG { 01:20:07 DEBUG "metric": { 01:20:07 DEBUG "pod": "ds-idrepo-1" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG [...] 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG "pod": "ds-cts-1" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 1713140407, 01:20:07 DEBUG "0" 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG } 01:20:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140407" 01:20:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140407" 01:20:07 INFO [http_cmd]: http status code OK 01:20:07 DEBUG --- status code --- 01:20:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:07 DEBUG --- http response --- 01:20:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:07 DEBUG ----- output ----- 01:20:07 DEBUG { 01:20:07 DEBUG "status": "success", 01:20:07 DEBUG "data": { 01:20:07 DEBUG "resultType": "vector", 01:20:07 DEBUG "result": [ 01:20:07 DEBUG { 01:20:07 DEBUG "metric": { 01:20:07 DEBUG "instance": "10.142.0.107:9100" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG [...] 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG "instance": "10.142.0.43:9100" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 1713140407, 01:20:07 DEBUG "8867.633333333335" 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG } 01:20:07 INFO [http_cmd]: http status code OK 01:20:07 DEBUG --- status code --- 01:20:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:07 DEBUG --- http response --- 01:20:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:07 DEBUG ----- output ----- 01:20:07 DEBUG { 01:20:07 DEBUG "status": "success", 01:20:07 DEBUG "data": { 01:20:07 DEBUG "resultType": "vector", 01:20:07 DEBUG "result": [ 01:20:07 DEBUG { 01:20:07 DEBUG "metric": { 01:20:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG [...] 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG 01:20:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:20:07 DEBUG }, 01:20:07 DEBUG "value": [ 01:20:07 DEBUG 1713140407, 01:20:07 DEBUG "0" 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG ] 01:20:07 DEBUG } 01:20:07 DEBUG } 01:20:25 INFO 01:20:25 INFO [loop_until]: kubectl --namespace=xlou top pods 01:20:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:20:25 INFO [loop_until]: OK (rc = 0) 01:20:25 DEBUG --- stdout --- 01:20:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 6m 8609Mi ds-cts-0 14m 8197Mi ds-cts-1 12m 8431Mi ds-cts-2 12m 8175Mi ds-idrepo-0 12m 4605Mi ds-idrepo-1 10m 4328Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1016Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 7m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1010m 6679Mi 01:20:25 DEBUG --- stderr --- 01:20:25 DEBUG 01:20:31 INFO 01:20:31 INFO [loop_until]: kubectl --namespace=xlou top node 01:20:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:20:31 INFO [loop_until]: OK (rc = 0) 01:20:31 DEBUG --- stdout --- 01:20:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4901Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 11068Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 10611Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 92m 0% 3091Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6852Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 12457Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2957Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 63m 0% 10130Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5981Mi 5% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10659Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2432Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 85m 0% 10204Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 6539Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 10554Mi 17% 01:20:31 DEBUG --- stderr --- 01:20:31 DEBUG 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 ERROR [http_cmd]: ERROR 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 ERROR --- status code --- 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 ERROR --- http response --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:21:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "ds-cts-2" 01:21:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:06 DEBUG }, 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0.0032054385270755513" 01:21:06 DEBUG "0.055" 01:21:06 DEBUG ] 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140466" 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "ds-idrepo-1" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "ds-idrepo-0" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG { 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "ds-cts-2" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG { 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "data": { 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG "result": [ 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG { 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "ds-idrepo-1" 01:21:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:21:06 DEBUG "pod": "ds-cts-2" 01:21:06 DEBUG }, 01:21:06 DEBUG }, 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG "value": [ 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG [...] 01:21:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "ds-idrepo-0" 01:21:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:21:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:06 DEBUG }, 01:21:06 DEBUG }, 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG "value": [ 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG "1.0313304901593674" 01:21:06 DEBUG "454.76347219964975" 01:21:06 DEBUG ] 01:21:06 DEBUG ] 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG ] 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "ds-idrepo-1" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "ds-idrepo-0" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140466" 01:21:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140466" 01:21:06 INFO [http_cmd]: http status code OK 01:21:06 DEBUG --- status code --- 01:21:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:06 DEBUG --- http response --- 01:21:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:06 DEBUG ----- output ----- 01:21:06 DEBUG { 01:21:06 DEBUG "status": "success", 01:21:06 DEBUG "data": { 01:21:06 DEBUG "resultType": "vector", 01:21:06 DEBUG "result": [ 01:21:06 DEBUG { 01:21:06 DEBUG "metric": { 01:21:06 DEBUG "pod": "ds-idrepo-1" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG [...] 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG 01:21:06 DEBUG "pod": "ds-idrepo-0" 01:21:06 DEBUG }, 01:21:06 DEBUG "value": [ 01:21:06 DEBUG 1713140466, 01:21:06 DEBUG "0" 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG ] 01:21:06 DEBUG } 01:21:06 DEBUG } 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "instance": "10.142.0.107:9100" 01:21:07 DEBUG "instance": "10.142.0.107:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "instance": "10.142.0.43:9100" 01:21:07 DEBUG "instance": "10.142.0.43:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140466, 01:21:07 DEBUG 1713140466, 01:21:07 DEBUG "0.00040000000000001514" 01:21:07 DEBUG "14395.466666666665" 01:21:07 DEBUG ] 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "instance": "10.142.0.107:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "instance": "10.142.0.43:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "13420391424" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140467" 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140467" 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "7114682368" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "pod": "ds-idrepo-1" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "pod": "ds-cts-1" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "0" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "0" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140467" 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "pod": "ds-idrepo-1" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "pod": "ds-idrepo-0" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "0" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "pod": "ds-cts-2" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "7149.517814255017" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "instance": "10.142.0.107:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "instance": "10.142.0.43:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "0.00033333333333333365" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "pod": "ds-idrepo-1" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "pod": "ds-cts-1" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "0" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "0" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140467" 01:21:07 INFO [http_cmd]: http status code OK 01:21:07 DEBUG --- status code --- 01:21:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:07 DEBUG --- http response --- 01:21:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:07 DEBUG ----- output ----- 01:21:07 DEBUG { 01:21:07 DEBUG "status": "success", 01:21:07 DEBUG "data": { 01:21:07 DEBUG "resultType": "vector", 01:21:07 DEBUG "result": [ 01:21:07 DEBUG { 01:21:07 DEBUG "metric": { 01:21:07 DEBUG "instance": "10.142.0.107:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG [...] 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG 01:21:07 DEBUG "instance": "10.142.0.43:9100" 01:21:07 DEBUG }, 01:21:07 DEBUG "value": [ 01:21:07 DEBUG 1713140467, 01:21:07 DEBUG "9230.633333333335" 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG ] 01:21:07 DEBUG } 01:21:07 DEBUG } 01:21:25 INFO 01:21:25 INFO [loop_until]: kubectl --namespace=xlou top pods 01:21:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:21:25 INFO [loop_until]: OK (rc = 0) 01:21:25 DEBUG --- stdout --- 01:21:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 6m 8610Mi ds-cts-0 13m 8197Mi ds-cts-1 12m 8431Mi ds-cts-2 13m 8173Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 11m 4328Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1016Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1019m 6704Mi 01:21:25 DEBUG --- stderr --- 01:21:25 DEBUG 01:21:31 INFO 01:21:31 INFO [loop_until]: kubectl --namespace=xlou top node 01:21:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:21:31 INFO [loop_until]: OK (rc = 0) 01:21:31 DEBUG --- stdout --- 01:21:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4904Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 11055Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10600Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3098Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 6838Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 12469Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10139Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5987Mi 5% gke-xlou-cdm-ds-0f601751-j488 67m 0% 10650Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2433Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 77m 0% 10199Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 6542Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1044m 6% 10579Mi 18% 01:21:31 DEBUG --- stderr --- 01:21:31 DEBUG 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140526" 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140526" 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140526" 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140526" 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140526" 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140526" 01:22:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140526" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- status code --- 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 ERROR [http_cmd]: ERROR 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 ERROR --- status code --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 ERROR --- http response --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG ----- output ----- 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:22:07 DEBUG { 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG { 01:22:07 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "data": { 01:22:07 DEBUG { 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG "data": { 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "data": { 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "data": { 01:22:07 DEBUG "data": { 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "data": { 01:22:07 DEBUG "data": { 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG { 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "result": [ 01:22:07 DEBUG "result": [ 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:22:07 DEBUG }, 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG "pod": "ds-cts-2" 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 DEBUG { 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "pod": "ds-cts-2" 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG }, 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG { 01:22:07 DEBUG 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG "data": { 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "result": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG { 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "metric": { 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG [...] 01:22:07 DEBUG [...] 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG }, 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140526, 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG "pod": "ds-idrepo-0" 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG "pod": "ds-idrepo-0" 01:22:07 DEBUG 1713140526, 01:22:07 DEBUG 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140526, 01:22:07 DEBUG 1713140526, 01:22:07 DEBUG "0" 01:22:07 DEBUG { 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG }, 01:22:07 DEBUG "1.0313413073060134" 01:22:07 DEBUG [...] 01:22:07 DEBUG 1713140526, 01:22:07 DEBUG "0" 01:22:07 DEBUG "0.055" 01:22:07 DEBUG ] 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "0" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG "value": [ 01:22:07 DEBUG ] 01:22:07 DEBUG 01:22:07 DEBUG "0.00001946403778716041" 01:22:07 DEBUG ] 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG "data": { 01:22:07 DEBUG ] 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG } 01:22:07 DEBUG 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG } 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG "0" 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG "result": [ 01:22:07 DEBUG ] 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG ] 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG "pod": "ds-idrepo-0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG { 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG }, 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "ds-cts-2" 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG ] 01:22:07 DEBUG ] 01:22:07 DEBUG "value": [ 01:22:07 DEBUG } 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG "0" 01:22:07 DEBUG 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG [...] 01:22:07 DEBUG ] 01:22:07 DEBUG 01:22:07 DEBUG } 01:22:07 DEBUG 01:22:07 DEBUG } 01:22:07 DEBUG 01:22:07 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "ds-idrepo-0" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "instance": "10.142.0.107:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "instance": "10.142.0.43:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0.0004666666666665264" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "instance": "10.142.0.107:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "instance": "10.142.0.43:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "14253.43333333333" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "ds-idrepo-0" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140527" 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "instance": "10.142.0.107:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "instance": "10.142.0.43:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "13421054976" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "7141457920" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "instance": "10.142.0.107:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "instance": "10.142.0.43:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "ds-cts-1" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "ds-cts-2" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "ds-idrepo-1" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "ds-cts-1" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "0" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140527" 01:22:07 INFO [http_cmd]: http status code OK 01:22:07 DEBUG --- status code --- 01:22:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:22:07 DEBUG --- http response --- 01:22:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:22:07 DEBUG ----- output ----- 01:22:07 DEBUG { 01:22:07 DEBUG "status": "success", 01:22:07 DEBUG "data": { 01:22:07 DEBUG "resultType": "vector", 01:22:07 DEBUG "result": [ 01:22:07 DEBUG { 01:22:07 DEBUG "metric": { 01:22:07 DEBUG "instance": "10.142.0.107:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG [...] 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG 01:22:07 DEBUG "instance": "10.142.0.43:9100" 01:22:07 DEBUG }, 01:22:07 DEBUG "value": [ 01:22:07 DEBUG 1713140527, 01:22:07 DEBUG "7706.9333333333325" 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG ] 01:22:07 DEBUG } 01:22:07 DEBUG } 01:22:25 INFO 01:22:25 INFO [loop_until]: kubectl --namespace=xlou top pods 01:22:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:22:25 INFO [loop_until]: OK (rc = 0) 01:22:25 DEBUG --- stdout --- 01:22:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8538Mi am-7d597cf7d6-8vr8f 7m 8661Mi am-7d597cf7d6-9wz9h 6m 8610Mi ds-cts-0 13m 8196Mi ds-cts-1 11m 8431Mi ds-cts-2 12m 8174Mi ds-idrepo-0 11m 4605Mi ds-idrepo-1 11m 4327Mi ds-idrepo-2 10m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 5m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1026m 6730Mi 01:22:25 DEBUG --- stderr --- 01:22:25 DEBUG 01:22:31 INFO 01:22:31 INFO [loop_until]: kubectl --namespace=xlou top node 01:22:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:22:31 INFO [loop_until]: OK (rc = 0) 01:22:31 DEBUG --- stdout --- 01:22:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 4905Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 11059Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10594Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3101Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6842Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 12487Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2956Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10134Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 5981Mi 5% gke-xlou-cdm-ds-0f601751-j488 62m 0% 10658Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2419Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 10192Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6528Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 989m 6% 10329Mi 17% 01:22:31 DEBUG --- stderr --- 01:22:31 DEBUG 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "ds-cts-2" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713140587" 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "1.045901822199234" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140587" 01:23:07 DEBUG --- status code --- 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:23:07 DEBUG { 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG "data": { 01:23:07 DEBUG { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG "data": { 01:23:07 DEBUG { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "result": [ 01:23:07 DEBUG "pod": "ds-cts-2" 01:23:07 DEBUG { 01:23:07 DEBUG }, 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG 01:23:07 DEBUG }, 01:23:07 DEBUG 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 01:23:07 DEBUG 01:23:07 DEBUG }, 01:23:07 DEBUG 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "pod": "ds-idrepo-0" 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG }, 01:23:07 DEBUG "0" 01:23:07 DEBUG "value": [ 01:23:07 DEBUG ] 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG } 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG --- status code --- 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 ERROR [http_cmd]: ERROR 01:23:07 DEBUG ] 01:23:07 DEBUG --- http response --- 01:23:07 ERROR --- status code --- 01:23:07 DEBUG } 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:23:07 DEBUG } 01:23:07 DEBUG ----- output ----- 01:23:07 ERROR --- http response --- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:23:07 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "instance": "10.142.0.107:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "instance": "10.142.0.43:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "14365.733333333332" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG { 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "ds-idrepo-0" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "ds-cts-2" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG { 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG --- http response --- 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG "data": { 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG { 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "metric": { 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:23:07 DEBUG "data": { 01:23:07 DEBUG "data": { 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG }, 01:23:07 DEBUG { 01:23:07 DEBUG "result": [ 01:23:07 DEBUG "result": [ 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG { 01:23:07 DEBUG { 01:23:07 DEBUG 01:23:07 DEBUG "data": { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG 01:23:07 DEBUG "result": [ 01:23:07 DEBUG }, 01:23:07 DEBUG }, 01:23:07 DEBUG [...] 01:23:07 DEBUG { 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG "metric": { 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG }, 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG "value": [ 01:23:07 DEBUG [...] 01:23:07 DEBUG [...] 01:23:07 DEBUG }, 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "0" 01:23:07 DEBUG [...] 01:23:07 DEBUG "pod": "ds-idrepo-0" 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG ] 01:23:07 DEBUG 01:23:07 DEBUG }, 01:23:07 DEBUG }, 01:23:07 DEBUG } 01:23:07 DEBUG 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "value": [ 01:23:07 DEBUG ] 01:23:07 DEBUG 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG } 01:23:07 DEBUG "pod": "ds-idrepo-0" 01:23:07 DEBUG "0" 01:23:07 DEBUG "0.055" 01:23:07 DEBUG } 01:23:07 DEBUG }, 01:23:07 DEBUG ] 01:23:07 DEBUG ] 01:23:07 DEBUG "value": [ 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG ] 01:23:07 DEBUG ] 01:23:07 DEBUG "0" 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG "metric": { 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG { 01:23:07 DEBUG }, 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "data": { 01:23:07 DEBUG 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG 01:23:07 DEBUG "result": [ 01:23:07 DEBUG 01:23:07 DEBUG { 01:23:07 DEBUG [...] 01:23:07 DEBUG "metric": { 01:23:07 DEBUG 01:23:07 DEBUG "instance": "10.142.0.107:9100" 01:23:07 DEBUG 01:23:07 DEBUG }, 01:23:07 DEBUG 01:23:07 DEBUG "value": [ 01:23:07 DEBUG "pod": "ds-idrepo-0" 01:23:07 DEBUG 01:23:07 DEBUG }, 01:23:07 DEBUG 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG [...] 01:23:07 DEBUG "0" 01:23:07 DEBUG 01:23:07 DEBUG ] 01:23:07 DEBUG 01:23:07 DEBUG } 01:23:07 DEBUG 01:23:07 DEBUG ] 01:23:07 DEBUG "instance": "10.142.0.43:9100" 01:23:07 DEBUG } 01:23:07 DEBUG }, 01:23:07 DEBUG } 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0.00040000000000001514" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713140587" 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "instance": "10.142.0.107:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "instance": "10.142.0.43:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "13421798400" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "7169671168" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "ds-cts-1" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "instance": "10.142.0.107:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "instance": "10.142.0.43:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0.00033333333333333365" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "ds-cts-2" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "ds-idrepo-1" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "ds-cts-1" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "pod": "am-7d597cf7d6-8ctft" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "pod": "am-7d597cf7d6-9wz9h" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "0" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:07 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713140587" 01:23:07 INFO [http_cmd]: http status code OK 01:23:07 DEBUG --- status code --- 01:23:07 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:23:07 DEBUG --- http response --- 01:23:07 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:23:07 DEBUG ----- output ----- 01:23:07 DEBUG { 01:23:07 DEBUG "status": "success", 01:23:07 DEBUG "data": { 01:23:07 DEBUG "resultType": "vector", 01:23:07 DEBUG "result": [ 01:23:07 DEBUG { 01:23:07 DEBUG "metric": { 01:23:07 DEBUG "instance": "10.142.0.107:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG [...] 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG 01:23:07 DEBUG "instance": "10.142.0.43:9100" 01:23:07 DEBUG }, 01:23:07 DEBUG "value": [ 01:23:07 DEBUG 1713140587, 01:23:07 DEBUG "7897.1" 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG ] 01:23:07 DEBUG } 01:23:07 DEBUG } 01:23:25 INFO 01:23:25 INFO [loop_until]: kubectl --namespace=xlou top pods 01:23:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:23:25 INFO [loop_until]: OK (rc = 0) 01:23:25 DEBUG --- stdout --- 01:23:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8538Mi am-7d597cf7d6-8vr8f 7m 8660Mi am-7d597cf7d6-9wz9h 6m 8609Mi ds-cts-0 13m 8197Mi ds-cts-1 12m 8430Mi ds-cts-2 12m 8175Mi ds-idrepo-0 11m 4606Mi ds-idrepo-1 11m 4327Mi ds-idrepo-2 11m 4100Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1016Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-5d8fc565dd-9g64c 6m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 529m 6751Mi 01:23:25 DEBUG --- stderr --- 01:23:25 DEBUG 01:23:31 INFO 01:23:31 INFO [loop_until]: kubectl --namespace=xlou top node 01:23:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:23:31 INFO [loop_until]: OK (rc = 0) 01:23:31 DEBUG --- stdout --- 01:23:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4902Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 11061Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 10587Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3092Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 64m 0% 6852Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 12512Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2949Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 65m 0% 10134Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 5993Mi 5% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10652Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2422Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10191Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6541Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 716m 4% 10660Mi 18% 01:23:31 DEBUG --- stderr --- 01:23:31 DEBUG 01:23:40 INFO Finished: True 01:23:40 INFO Waiting for threads to register finish flag 01:24:31 INFO Done. Have a nice day! :) 127.0.0.1 - - [15/Apr/2024 01:24:31] "GET /monitoring/stop HTTP/1.1" 200 - 01:24:35 WARNING File /tmp/lodemon_data-20240414_182729/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6511', ...) 127.0.0.1 - - [15/Apr/2024 01:24:40] "GET /monitoring/process HTTP/1.1" 200 -